2021-06-30 retrospective & plan discussion

[kgodey]

Retrospective

This covers work since our last discussion on 2021-06-23.

Product

• Issues created to track:
  • Eventual logo & branding for Mathesar: Create logo and branding for Mathesar #286
  • Better design for import status indicators: Design for import status indicators #298

Design

• Finalized review and design for 4. Editable Tables milestone
• Started design for 5. Multiple Database Support
• Improvements to Design Spec document format

Development

• Frontend work on 1. Viewing Tables (Milestone closed)
  • Table Pagination completed
  • Search option in sidebar
  • Stale requests are now aborted
  • Invalid table ids in url is handled
• Backend work on 2. Tables for File Import milestone:
  • Table data types inference endpoint completed
  • Additional table filtering options completed
  • Duplicate test database issue fixed
  • Engine caching issue raised: Schema sqlalchemy engine caching is per-instance #280
  • Column API work nearly complete
• Backend work on 3. Basic Filtering, Sorting, Grouping in progress:
  • Sorting, filtering and grouping records
  • Creating usable fork of sqlalchemy-filters

Infrastructure

• Continued work on staging server setup.
  • Mathesar settings were refactored to allow for multiple deployments
  • Ansible playbook work is ongoing

Active Work

This is work planned for the upcoming week.

Product

• @kgodey: Create additional issues for backend work on 5. Multiple Database Support.
• @kgodey: Review of milestones 2-8 and ensure that issues created cover all work, since new concerns have come up in the last week.

Design

• @ghislaineguerin: Improvement to design for 2. Tables for File Import milestone:
  • Move table creation outside of modal
• @ghislaineguerin: Design review and spec finalization for 5. Multiple Database Support design.
• @ghislaineguerin: Start design work for 6. Working with Schemas

Development

• Backend work on 2. Tables for File Import milestone needs to be completed:
  • @mathemancer: Need to spec and implement confirm data API endpoint
    • Need to create a way to show a preview of data. Since we're no longer actually altering the table when inferring types, this implies creating some kind of preview table
    • We also need to be able to use the first row as header or not, which implies re-importing the data under those different conditions
  • @mathemancer: Need to complete all other endpoints needed for the file import backend.
  • @mathemancer: Caching for SQLAlchemy engine objects.
  • @eito-fis: Remove legacy CSV functions once frontend no longer uses them.
• @eito-fis Backend work on 3. Basic Filtering, Sorting, Grouping:
  • Sorting, filtering and grouping records
  • Creating usable fork of sqlalchemy-filters
• @pavish Frontend work on 2. Tables for File Import milestone
  • Setup storybook
  • Handle documentation and tests for Button, TextInput, Icon and Skeleton
  • File upload component
  • File import pages, switch to new CSV import API
• @eito-fis: Backend work for 5. Multiple Database Support design.
  • Create databases API that show supported types.
  • Support configuration of multiple databases in the backend.
  • Ensure database objects are imported when new database is connected.
• @eito-fis: Start backend work for 6. Working with Schemas
  • DB functions for renaming and deleting schemas.

Infrastructure

• @kgodey: Complete staging server setup (might need @pavish's help for some frontend build work).
• @kgodey: Set up deployment for Storybook

Updates and Questions

• We completed our first milestone, 1. Viewing Tables on time! Nice work, everyone.
• I'm moving the deadline for 2. Tables for File Import out a week to July 12 because backend and design work is ongoing.
  • Please note that 3. Basic Filtering, Sorting, Grouping is also scheduled for July 12. Backend work for the milestone is almost complete, so we should be able to do both by July 12.
• I'll go ahead and move everything listed under active work above to the Active sprint by the end of the day.
• Questions:
  • How did this week go for everyone?
    • What went well?
    • Anything that could be improved?
  • Do you anticipate any blockers next week?
  • Any questions or comments on any of the above?

[pavish]

What went well?

We completed the milestone, work is progressing as expected. Overall, I'd say everything went well.

Anything that could be improved?

The frontend tests for Milestone 1 couldn't be completed by the due date, and moved to Milestone 2. This is because of us requiring to create new components in the initial milestones, which may lead to more frontend tickets per milestone and more time than anticipated. We may see some delay in the first few milestones, and would have to take that into account.

Milestone 2 currently has 7 frontend tickets. I'm estimating frontend work for Milestone 2 to complete by 6th or 7th July. I'm glad the deadline is moved to 12th. However, Milestone 3 has a lot of work on the frontend, and may get delayed.

Do you anticipate any blockers next week?

No blockers. May have questions on file import API changes.

Any questions or comments on any of the above?

None.

[kgodey]

@pavish we'll check in on Milestone 3's deadline during next week's discussion.

[eito-fis]

What went well?

Milestone completed! Also, getting feedback on the type inference work and record filtering decisions was nice.

Anything that could be improved?

Nothing specific comes to mind.

Do you anticipate any blockers next week?

Will need some additional details before starting record grouping work.

Any questions or comments on any of the above?

None.

[mathemancer]

• How did this week go for everyone?
  • What went well?
    The overall cohesiveness of the back end seems to be improving. I'm really happy that we're finally getting to a place where we can use the design and front-end needs to inform back end work.
  • Anything that could be improved?
    I went in circles a bit on the Column API implementation. I'm not sure that there's anything specific to improve there, but it felt unproductive at times.
• Do you anticipate any blockers next week?
  We'll need to coordinate a bit on the API requirements for the confirm data endpoint. There could be delays due to TZ issues (i.e., waiting to hear back on questions, etc.)
• Any questions or comments on any of the above?
  Nope!

[kgodey]

@mathemancer would speccing the Column API out ahead of time have helped?

Please try and create as complete a plan as possible for the confirm data endpoint before we talk so that we can minimize back and forth. I'll prioritize answering questions as well.

[mathemancer]

I'm not sure it would have helped, since most of the back-and-forth was due to things discovered while implementing. And the actual endpoint didn't change behaviorally, just the implementation.

For the confirm data import, I'll definitely spec out the endpoint behavior I'm thinking before starting implementation.

[kgodey]

That makes sense. I consider implementation details a part of technical specs, FWIW.

[mathemancer]

I think it's definitely a good idea to record them somewhere (the thinking behind the implementation details, anyway) in the case of an open project like this, and the spec seems like a good place. I do think that when trying a different implementation options, it makes sense to try them, and then record the one that ends up working best.

[kgodey]

What went well?

We're getting into a good cadence of work. We completed our first milestone, which is great. Design is a few milestones ahead, and backend is getting there.

Anything that could be improved?

Personally, I need to balance code work and non-code work better, I need to do a lot of different types of work and it's been harder than expected to find good stretches of time to focus on code work. I'll continue iterating on that.

Do you anticipate any blockers next week?

• I'll need to coordinate with @pavish on the frontend build steps to get staging up.
• I'll need to discuss the backend APIs for file import with @mathemancer, they're still not fully defined.

Any questions or comments on any of the above?

No, I'll comment on individual people's responses with any questions.

[ghislaineguerin]

How did this week go for everyone?
What went well?

It was a good week. I can feel how the simplified design specs allow me to submit things to review sooner. Feels much more manageable to embed the Figma prototypes rather than static screenshots.

Anything that could be improved?

I could communicate better when I introduce future features into my prototypes and be more clear about the reasons for doing so (anticipating needs, etc.)

Do you anticipate any blockers next week?

No.

Any questions or comments on any of the above?

Not for now.