SB 5.1 - OCI - "Last updated on" is showing as invalid date for admin reports #481

subhashchandrab · 2023-05-09T04:56:54Z

subhashchandrab
May 9, 2023

Login with Report admin credentials
Go to the admin dashboard
Click on existing reports
Observe the ‘Last Updated on’ status

Invalid date’ text is showing in all reports of the admin dashboard

Though we are able to view the data in the report, the filed "Last updated on" is showing as Invalid date in the UI. All the required buckets for storing the reports are configured in the backend.

We would like to understand where this data(Last updated on) will be fetched from?

subhashchandrab · 2023-05-09T04:58:19Z

subhashchandrab
May 9, 2023
Author

0 replies

chitranshu-keshav · 2023-05-11T03:48:43Z

chitranshu-keshav
May 11, 2023
Collaborator

@rajeevsathish @vinukumar-vs Can you please get back on the above

0 replies

rajeshkumaravel · 2023-05-11T05:51:03Z

rajeshkumaravel
May 11, 2023

@subhashchandrab
Please verify the response from /course-reports/metadata?fileNames= API
The expected value from response key result[REPORT_NAME]. lastModified should be in format "Fri, 27 Jan 2023 05:29:42 GMT"
P.S: If key (or) value is not in expected format; then it might be data issue

CC: @Krishnaj20 @vinukumar-vs @rajeevsathish

4 replies

subhashchandrab May 11, 2023
Author

@rajeshkumaravel The following is the response for the API invocation. I don't see any field like lastModified in the response
API - https://diksha.gov.in/course-reports/metadata?fileNames=%7B%22ecg_nation_learning%22:%22public/ecg_nation_learning.json%22%7D

{"id":"api.report","ver":"1.0","ts":"2023-05-11 06:36:44:823+0000","params":{"resmsgid":"3280fa70-efc6-11ed-8478-a346a15a031a","msgid":null,"status":"success","err":null,"errmsg":null},"responseCode":"OK","result":{"ecg_nation_learning":{"reportname":"ecg_nation_learning","statusCode":200}}}

However, there is another API invocation as mentioned below
https://diksha.gov.in/reports/fetch/public/ecg_nation_learning.json

The response for this API contains all the data of the corresponding report and also it has the following field in the response json
metadata: {lastUpdatedOn: 1585827673630.767}

This doesn't look like a valid date/timestamp. This field seems to be specific to this API alone.

subhashchandrab May 11, 2023
Author

@rajeshkumaravel Can you please confirm which service we should look into for identifying this logic which is fetching the lastUpdatedOn field

vinukumar-vs May 12, 2023

@anandp504 Is this property "lastModified" is adding by report-service?
Endpoint "{{host}}/course-reports/metadata".

Is the response only below properties present.

"result": {
        "ecg_nation_learning": {
            "reportname": "ecg_nation_learning",
            "statusCode": 200
        }
    }

If it is adding by the report service, what can be the reason for not returning this value?
@subhashchandrab @Krishnaj20 @gandham-santhosh

vinukumar-vs May 15, 2023

@anandp504 @manjudr @reshmi-nair Who this property lastModified being added to the response?

API is not returning the response, hence it is not showing the lastUpdatedOn in the UI. Help @subhashchandrab how can they debug the issue with OCI infra.

@Krishnaj20 @rhwarrier @anandvarada

vinukumar-vs · 2023-05-12T07:50:04Z

vinukumar-vs
May 12, 2023

@subhashchandrab report-service still doesn't support for cloud agnostic(for few reports), hence the below features will be impacted.

Program reports showing for Admin (JIRA story: https://project-sunbird.atlassian.net/browse/SB-28247)

@anandp504 @Krishnaj20 @rhwarrier

2 replies

subhashchandrab May 12, 2023
Author

As I understand, this is related to applying filters on top of Program Dashboard reports. In general, the Program Dashboard report request, generation and viewing is happening fine in Diksha currently. Any other impact apart from the above mentioned area.

Krishnaj20 May 16, 2023
Maintainer

@subhashchandrab report-service still doesn't support for cloud agnostic(for few reports), hence the below features will be impacted.

Program reports showing for Admin (JIRA story: https://project-sunbird.atlassian.net/browse/SB-28247)

@anandp504 @Krishnaj20 @rhwarrier

@SanthoshVasabhaktula - tagging you to this thread. As per the initial analysis the team has confirmed that the report service is not yet cloud agnostic.

ddevadat · 2023-05-17T13:03:02Z

ddevadat
May 17, 2023

@vinukumar-vs any progress on this issue?

2 replies

vinukumar-vs May 17, 2023

@anandp504 Please confirm once the PR is merged & tested.
project-sunbird/sunbird-report-service#40

@anandp504 @Krishnaj20 @SanthoshVasabhaktula

ddevadat May 18, 2023

@vinukumar-vs @anandp504
any eta on this. Meanwhile i tried to taken in the changes from the PR to see if it works..build goes fine..but the report pod doesnt comes up.

jenkins@jenkins:~$ k logs -f report-76d59b6698-7h7f7

> hawkeye-server@1.0.0 start /opt/report
> cross-env DEBUG=* node ./bin/www "&"

Thu, 18 May 2023 06:22:24 GMT report-service:server { use_env_variable: true,
  dialect: 'postgres',
  dialectOptions: { ssl: { rejectUnauthorized: false } } }
2023-05-18T06:22:24.065Z sequelize:pool pool created with max/min: 5/0, no replication
2023-05-18T06:22:24.090Z sequelize:hooks adding hook beforeValidate
2023-05-18T06:22:24.090Z sequelize:hooks adding hook beforeUpdate
/opt/report/node_modules/client-cloud-services/dist/bundle.js:2
(()=>{var __webpack_modules__={66882:(e,t,r)=>{"use strict";Object.defineProperty(t,"__esModule",{value:!0}),t.AwsCrc32=void 0;var n=r(38570),a=r(61173),i=r(43421),o=function(){function e(){this.crc32=new i.Crc32}return e.prototype.update=function(e){(0,a.isEmptyData)(e)||this.crc32.update((0,a.convertToBuffer)(e))},e.prototype.digest=function(){return(0,n.__awaiter)(this,void 0,void 0,(function(){return(0,n.__generator)(this,(function(e){return[2,(0,a.numToUint8)(this.crc32.digest())]}))}))},e}();t.AwsCrc32=o},43421:(e,t,r)=>{"use strict";Object.defineProperty(t,"__esModule",{value:!0}),t.AwsCrc32=t.Crc32=t.crc32=void 0;var n=r(38570),a=r(61173);t.crc32=function(e){return(new i).update(e).digest()};var i=function(){function e(){this.checksum=4294967295}return e.prototype.update=function(e){var t,r;try{for(var a=(0,n.__values)(e),i=a.next();!i.done;i=a.next()){var s=i.value;this.checksum=this.checksum>>>8^o[255&(this.checksum^s)]}}catch(e){t={e

SyntaxError: Unexpected reserved word
    at createScript (vm.js:80:10)
    at Object.runInThisContext (vm.js:139:10)
    at Module._compile (module.js:616:28)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
    at Module.require (module.js:596:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/opt/report/helpers/cloudStorageServices/index.js:12:23)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! hawkeye-server@1.0.0 start: `cross-env DEBUG=* node ./bin/www "&"`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the hawkeye-server@1.0.0 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2023-05-18T06_22_24_410Z-debug.log

ddevadat · 2023-05-18T07:48:59Z

ddevadat
May 18, 2023

@rajeshkumaravel

We have a version of client cloud service which has oci support

https://github.com/ocisunbird/client-cloud-services/tree/oci-5.1.0

is it possible to merge this into the upstream branch , so that its packaged as an npm package.

Rightnow the PR project-sunbird/sunbird-report-service#40
uses version: 2.0.3, which will not support OCI.

Once we have a new version say 2.0.6, we can use that it should support oci

Cc: @vinukumar-vs @Krishnaj20 @anandp504 @santhosh-tg

2 replies

vinukumar-vs May 22, 2023

@ddevadat Can you contribute(GIT PR) the OCI related changes? We will merge the & publish the NPM package.

@pallakartheekreddy has shared the PR for report-service related changes in the Client-Cloud-Service library.
Sunbird-Ed/client-cloud-services#13

Once we publish these changes, he will integrate Client-Cloud-Service NPM package with report-service.
@ddevadat is you share OCI integration PR, we can publish the library together.

@rajeshkumaravel @rajeevsathish @Krishnaj20 @anandvarada

vinukumar-vs May 22, 2023

@pallakartheekreddy Client-cloud-services NPM library is published with your PR changes. You can Integrate into Report-service now.
https://www.npmjs.com/package/client-cloud-services/v/2.0.6

@@swayangjit @rajeshkumaravel @rajeevsathish @Krishnaj20 @anandvarada @ddevadat

ddevadat · 2023-05-19T00:33:06Z

ddevadat
May 19, 2023

@vinukumar-vs @rajeshkumaravel @anandp504

I think the problem here is the AWS implementation of client cloud service code.
https://github.com/Sunbird-Ed/client-cloud-services/blob/main/src/AWSStorageService.js

Because oci is an s3 compliant api storage, we used the existing AWS implementation for customization. The assumption here was that aws implementation is working fine.

The problem was in the method

  async fileExists(bucketName, fileToGet, prefix = '', cb) {
    const params = { Bucket: bucketName, Key: prefix + fileToGet };
    const command = new HeadObjectCommand(params);
    logger.info({ msg: 'AWS__StorageService - fileExists called for bucketName ' + bucketName + ' for file ' + params.Key });
    await this.client.send(command).then((resp) => {
      cb(null, resp)
    }).catch((err) => {
      cb(err);
    });
  }

It uses HeadObjectCommand method from aws lib
https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/classes/headobjectcommand.html

Two of the output attributes will be
LastModified
ContentLength

For the azure counterpart, it uses https://azure.github.io/azure-storage-node/BlobResult.html
lastModified
contentLength

Notice the lower case.

In the getFileProperties(), the response o/p is created using the lower case

        async.parallel(getBlogRequest, (err, results) => {
          if (results) {
            results.forEach(blob => {
              if (blob.error) {
                responseData[(_.get(blob, 'error.reportname'))] = blob.error
              } else {
                responseData[(_.get(blob, 'value.reportname'))] = {
                  lastModified: _.get(blob, 'value.lastModified'),
                  reportname: _.get(blob, 'value.reportname'),
                  statusCode: _.get(blob, 'value.statusCode'),
                  fileSize: _.get(blob, 'value.contentLength')
                }
              }
            });

This lowercase value of lastModified & contentLength will not be there in the aws s3 response.

we corrected this in the oci implementation
https://github.com/ocisunbird/client-cloud-services/blob/oci-5.1.0/src/OCIStorageService.js

        async.parallel(getBlogRequest, (err, results) => {
          if (results) {
            results.forEach(blob => {
              if (blob.error) {
                responseData[(_.get(blob, 'error.reportname'))] = blob.error
              } else {
                responseData[(_.get(blob, 'value.reportname'))] = {
                  lastModified: _.get(blob, 'value.LastModified'),
                  reportname: _.get(blob, 'value.reportname'),
                  statusCode: _.get(blob, 'value.statusCode'),
                  fileSize: _.get(blob, 'value.ContentLength')
                }
              }
            });

Now the last updated date is showing, but we have a potential problem.

The reponse data is

{"id":"api.report","ver":"1.0","ts":"2023-05-18 23:49:31:516+0000","params":{"resmsgid":"a26c33c0-f5d6-11ed-a994-6b81464953d4","msgid":null,"status":"success","err":null,"errmsg":null},"responseCode":"OK","result":{"total_content_plays_on_diksha_app_and_portal":{"lastModified":"2023-05-08T05:13:44.000Z","reportname":"total_content_plays_on_diksha_app_and_portal","statusCode":200,"fileSize":1679}}}

Is this the expected output or do need to convert the response date from 2023-05-08T05:13:44.000Z to Mon, 8 May 2023 05:13:44 GMT

6 replies

This comment was marked as off-topic.

Sign in to view

ddevadat May 22, 2023

@ddevadat Can you contribute(GIT PR) the OCI related changes? We will merge the & publish the NPM package.

@pallakartheekreddy has shared the PR for report-service related changes in the Client-Cloud-Service library. Sunbird-Ed/client-cloud-services#13

Once we publish these changes, he will integrate Client-Cloud-Service NPM package with report-service. @ddevadat is you share OCI integration PR, we can publish the library together.

@rajeshkumaravel @rajeevsathish @Krishnaj20 @anandvarada

@vinukumar-vs
Created PR for merging oci implementation
Sunbird-Ed/client-cloud-services#15

Please let us know when the new npm package is released

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sunbird-ED (Sunbird for Education)

SB 5.1 - OCI - "Last updated on" is showing as invalid date for admin reports #481

{{title}}

Replies: 7 comments 16 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

This comment was marked as off-topic.

{{title}}

Select a reply

SB 5.1 - OCI - "Last updated on" is showing as invalid date for admin reports #481

Replies: 7 comments · 16 replies

subhashchandrab May 9, 2023 Author

chitranshu-keshav May 11, 2023 Collaborator

subhashchandrab May 11, 2023 Author

subhashchandrab May 11, 2023 Author

subhashchandrab May 12, 2023 Author

Krishnaj20 May 16, 2023 Maintainer

This comment was marked as off-topic.

Replies: 7 comments 16 replies

subhashchandrab
May 9, 2023
Author

chitranshu-keshav
May 11, 2023
Collaborator

subhashchandrab May 11, 2023
Author

subhashchandrab May 11, 2023
Author

subhashchandrab May 12, 2023
Author

Krishnaj20 May 16, 2023
Maintainer