Making the frontend type-agnostic

[pavish]

Types are currently hard-coded at the component level on the frontend. The eventual goal is to make the frontend type-agnostic. Inorder to achieve that, the following information has to be provided by the backend: (I'm only covering the functional parts here and not the presentational elements)

• Cell Input
  • How the cell for each type should accept, validate, display user input
• Filters
  • A set of filter conditions and accepted parameters for those conditions
• Groups
  • Allowed grouping options for each mathesar_type, and how to construct the UI for each grouping option
• Type configurations when user updates type of a column
  • Database configuration for each type. eg., Text gets input for field size, number gets sub type and additional config etc.,
  • Display configuration for each type. eg., Number has display options for 'Show as percentage' and format.
  • Determination of db_type based on the selected mathesar_type and user specified type configuration.
  • UX Reference: Figma prototype

Related:

• Store icon, labels, and input options for types on the backend #661
• Extending the filtering API #848

---

If we were to hard-code all of these at the component layer, it will involve a massive refactor when we move to the api at a later stage.

Inorder to avoid that, we decided to move all the type related configurations into the data layer at the frontend. That way the component layer remains type-agnostic whereas the data layer has an isolated module which deals with hardcoded type configurations which would have the same response as we would expect the api to provide. When we are ready, the module could be modified to fetch the configuration data from an API without much hassle.

I've encountered some issues when implementing this, which would also be a factor when we move to the API. The following points are in the perspective of changing a column's type.

Approach 0 (Current approach): Use a seriazliable JSON to represent type options, which can be swapped by the API response.

As mentioned above, this will involve maintaining a hard-coded JSON on the frontend, which would later be provided by the API.

1. Some aspects of our UX is a bit complex to represent using a serializable JSON, although it is possible.
   eg.,
   The UI for Number type:

   

   Requires the following JSON representation:

   {
     ...,
     [ABSTRACT_TYPE_CATEGORIES.Number]: {
       typeOptions: {
         database: {
           allowDefault: true,
           inputs: [
             {
               type: 'select',
               id: 'numberType',
               label: 'Number Type',
               options: ['Integer', 'Decimal', 'Float'],
             },
             {
               type: 'select',
               id: 'integerDataSize',
               label: 'Integer Data Size',
               options: ['Long int', 'Small int'],
               showOn: {
                 id: 'numberType',
                 op: 'eq',
                 value: 'Integer',
               },
             },
             {
               type: 'number',
               id: 'decimalPlaces',
               label: 'Decimal Places',
               showOn: {
                 id: 'numberType',
                 op: 'eq',
                 value: 'Decimal',
               },
             },
             {
               type: 'number',
               id: 'maxDigits',
               label: 'Max Digits',
               showOn: {
                 id: 'maxDigits',
                 op: 'eq',
                 value: 'Decimal',
               },
             },
             ...,
           ],
         },
         display: {
           inputs: [
             {
               type: 'toggle',
               id: 'showAsPercentage',
               label: 'Show as Percentage',
             },
             {
               type: 'select',
               id: 'format',
               label: 'Format',
               options: [
                 {
                   id: 1,
                   label: 'English (US)'
                 },
                 {
                   id: 2,
                   label: 'French (FR)'
                 }
               ]
             }
           ]
         }
       },
     },
     ...,
   }

   This would be a pain to maintain in the long run.

2. The frontend handles determination of the db_type from the selected mathesar_type + user configurations. This requires letting the frontend know how to do that.

   • Inorder for the frontend to know how to determine the db_type, it requires additional information.
     eg.,

      {
        ...,
        [ABSTRACT_TYPE_CATEGORIES.Text]: {
          typeOptions: {
            database: {
              allowDefault: true,
              fields: [
                {
                  type: 'toggle',
                  id: 'restrictFieldSize',
                  label: 'Restrict Field Size'
                },
                {
                  type: 'number',
                  id: 'fieldSizeLimit',
                  label: 'Field Size Limit',
                  showOn: {
                    id: 'restrictFieldSize',
                    op: 'eq',
                    value: true,
                  }
                }
              ],
              determinationRules: {
                'CHAR': {
                  'and': [
                    {
                      id: 'restrictFieldSize',
                      op: 'eq',
                      value: true,
                    },
                    {
                      id: 'fieldSizeLimit',
                      op: 'lte',
                      value: 255
                    }
                  ]
                },
                'VARCHAR': {
                  'and': [
                    {
                      id: 'restrictFieldSize',
                      op: 'eq',
                      value: true,
                    },
                    {
                      id: 'fieldSizeLimit',
                      op: 'lte',
                      value: 32672
                    }
                  ]
                },
                'TEXT': {
                  id: 'restrictFieldSize',
                  op: 'eq',
                  value: false,
                }
              }
            },
          },
        },
        ...,
      }

     Refer the determinationRules property.
   • Another option is to let the backend do it. Pass the user input to an api and it would return the db_type.

3. Validation & Display of user input

   • The validation and display of input entered by the user cannot be done strictly based on the mathesar_type. Each db_type has different characteristics even if they belong to the same mathesar type.
     • eg., The Decimal type requires users to enter a decimal point and to display it based on the display configurations, whereas an integer type should not allow it. However, both of these are considered to be a Number mathesar_type.
   • This would require the frontend to be aware of each database type.
   • We may need to re-evaluate what separates a mathesar_type from a db_type and where the validation part lies. This is related to Create Concepts page for data types mathesar-wiki#14.

All of these lead me to accept that it is going to be incredibly hard to do this without implementing dedicated frontend code for each type.

I came up with some alternative approaches:

Alternative approach 1: Implement types as plugins on the frontend.

• We could implement UI plugins which are isolated from the rest of the code and loaded dynamically on the frontend.
• The API would return the plugin associated with each type, and the frontend will load that plugin.
• This would handle UI layout construction, validation, display and determination of types, natively as frontend code.
• This would not require the API to send UI construction related information.

Drawbacks

• It would be a lot of work and we cannot do it for the initial release.
  • We can try to implement a stripped down version for alpha release. It would load the plugin code directly instead of dynamic load, which should be extensible enough for us to later make it dynamic.
• It will lead to security vulnerabilities. [BLOCKER]
  • I had this plugin approach in mind to implement features like the Calendar and Kanban views, which are more isolated. They can be rendered within an iframe (and preferably using a different port) thereby sandboxing them from the rest of the app.
  • Type configurations on the other hand, cannot be sandboxed. They are displayed as part of the core layout and it needs to be rendered there as well. This will lead to the code having access to everything on the frontend including cookies and localstorage. This is not desirable and might lead to exploitation.

Alternative approach 2: Re-evaluate mathesar_types and make UX for types simple enough to be represented as JSON

• Our current UX has a lot of elements which are shown/hidden based on other elements. They also have additional configurations based on pre-selected options of other inputs.
• This leads to the JSON representation being complex to construct, as mentioned in above examples.
• We can split mathesar_types to only include types that are more similar. eg., Split Integer, Big Int, Small Int as a separate category and Decimal, Float as a different category.
• We can simplify the UX to make it easier to represent as JSON.
• This will remove complexity of the current approach.

Drawbacks

• The current UX for type switching is really good and I personally do not want to abandon it.

Alternative approach 3: Support JSON representation but override it for known types we provide with the product

• For known types (ones that we have UX for), we can hardcode configurations on the frontend.
• For unknown types, we can rely on the api response for layout construction and the rest.
• This is the same as current approach, except known types are exempt from it.
• This will retain the current UX for type switching for known types, and have a simplified UX for unknown types.

Drawbacks

• Splits types into known and unknown and overrides API response for known types. This makes it harder to maintain it uniformly on the backend.
• The frontend will not be truly type-agnostic, but it will be able to support unknown types, making it partially-type-agnostic.

I am inclined to go with Alternative approach 3.

I would love to hear your thoughts on this. @kgodey @seancolsen @mathemancer @dmos62 @silentninja

[seancolsen]

"Alternative approach 3" sounds good for now. @pavish

I really like "Alternative approach 1: Implement types as plugins on the frontend", and would hope that we could move in that direction in the long run. Obviously it raises many other questions, including security, so probably not worth tackling for the alpha release.

[pavish]

Implementing frontend plugins is what I'd like us to go with for Graphs, Forms, Kanban and Calendar etc., They are more isolated and we can ensure proper security. However, type switching is too attached to the core layout and we cannot sandbox it. We can have a separate discussion after our alpha for the plugin system as a whole.

[dmos62]

I think this relies a lot on what happens with Mathesar types (mathesar-foundation/mathesar-wiki#14). I feel like a combination of improving Mathesar type definition and simplifying UX would solve this in a robust way. I too like the current UX and I think that we don't have to discard it, but it might need iteration to account for the UI's declarative construction. Again, #14 will be at the center of this.

Approach 1, serving frontend modules/code through the API, does not seem useful to me, off the top of my head. The modularization does not help with defining or using the API in a declarative way, right? Or, is this a way to achieve portability and scalability without declarativeness? Currently, I'm not convinced this would be effective.

I think that we're on the right path and some iteration on Mathesar types and UX will be effective at addressing this. Procedural UI construction is a really cool problem and if we solve this right it could be a significant selling-point for Mathesar, I think.

Edit: by the way, I really liked your samples from the frontend's data layer. They capture the complexity of the various operations and data structures very well, in my opinion. You've said that you see these as being hard to maintain, but a non-declarative UI would be much worse, right? The complexity seems to come from the complexity of database and Mathesar types, which I'd call essential complexity in this context. Again, I think the declarations you shared are on really good track.

[pavish]

Procedural UI construction is a really cool problem and if we solve this right it could be a significant selling-point for Mathesar

We would require procedural UI construction to provide user defined graphs and dashboards (if we intend to support them in the future). I'm not sure how good of a selling-point it would be, but a cool problem to solve, none the less.

In the discussion with Approach 3, I'm not mentioning completely removing procedural UI construction for known types. We would have it but add custom additional functionality on the frontend overriding some aspects of it to be able to construct the UX accurately.

[dmos62]

To expand on rationale behind me saying that procedural UI is a selling point is that it's a strong sign of good abstractions, which is a strong sign of a good tool. Though I can see other perspectives on this.

[kgodey]

I support Alternative Approach 3 with one caveat. Could we defer JSON representation of unknown types to the future, when we do our plugin architecture milestone? That way, we'll know more about how we're handling types and how we expect to plug new types in.

As far as Alternative Approach 2, I'm fine with reevaluating some Mathesar types, but I don't think we should do it on the level of Integer vs. Decimal being separate types, this is not as approachable for non-technical users.

I think you covered the drawbacks of Alternative Approach 1 pretty well.

[pavish]

I don't think we should do it on the level of Integer vs. Decimal being separate types, this is not as approachable for non-technical users.

Agreed. I'll come up with a declarative format that handles this.

I support Alternative Approach 3 with one caveat. Could we defer JSON representation of unknown types to the future, when we do our plugin architecture milestone?

I don't think we can completely defer the JSON representation since we need to place this on the data layer on the frontend. We have to represent it for the known types at least.

The idea is to have the JSON but override some aspects of it on the frontend for known types (eg., adding some custom css classes based on hardcoded check on the frontend).

However, we do not have to worry about the unknown types for now.

[kgodey]

Okay, thanks. Please open a new discussion with the final declarative format that you come up with so people can provide feedback if needed.