Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lakectl fs use sync manger #6650

Merged
merged 16 commits into from
Oct 1, 2023
Merged

lakectl fs use sync manger #6650

merged 16 commits into from
Oct 1, 2023

Conversation

idanovo
Copy link
Contributor

@idanovo idanovo commented Sep 26, 2023

Closes #6321

@idanovo idanovo added area/lakectl Issues related to lakeFS' command line interface (lakectl) include-changelog PR description should be included in next release changelog v1.0.0-blocker Issues that should be closed before going out with v1.0.0 labels Sep 26, 2023
@idanovo idanovo self-assigned this Sep 26, 2023
@idanovo idanovo changed the title New line lakectl fs use sync manger Sep 26, 2023
@idanovo idanovo requested a review from N-o-Z September 26, 2023 15:52
)

var fsDownloadCmd = &cobra.Command{
Use: "download <path uri> [<destination path>]",
Short: "Download object(s) from a given repository path",
Args: cobra.RangeArgs(fsDownloadCmdMinArgs, fsDownloadCmdMaxArgs),
Run: func(cmd *cobra.Command, args []string) {
pathURI := MustParsePathURI("path", args[0])
remote, dest := getLocalArgs(args, true, false)
flagSet := cmd.Flags()
preSignMode := Must(flagSet.GetBool("pre-sign"))
Copy link
Member

@N-o-Z N-o-Z Sep 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can use getLocalSyncFlags for the presign and parallelism flags (Though I would remove the Local from its name)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can't use it as it uses pre-signed as true by default

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might want to change that to be true by default, maybe @ozkatz can comment on that

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it will be a better thing to do it in a different PR especially because this issue

Copy link
Member

@N-o-Z N-o-Z left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good.
I'd like to raise 2 fundamental issues which might require a product discussion:

  1. We might want to consider consolidating the single / recursive flows and use the sync manager in both + removing the recursive flag
  2. We should take advantage of the parallelism of the sync manager and introduce this parameter in the upload command + use the default value
  3. The sync manager provides a summary of the performed operations - we should use this as the output for the commands (you can see examples of usage in the local commands)

)

var fsDownloadCmd = &cobra.Command{
Use: "download <path uri> [<destination path>]",
Short: "Download object(s) from a given repository path",
Args: cobra.RangeArgs(fsDownloadCmdMinArgs, fsDownloadCmdMaxArgs),
Run: func(cmd *cobra.Command, args []string) {
pathURI := MustParsePathURI("path", args[0])
remote, dest := getLocalArgs(args, true, false)
flagSet := cmd.Flags()
preSignMode := Must(flagSet.GetBool("pre-sign"))
recursive := Must(flagSet.GetBool("recursive"))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also we might want to move the parallel check ( <1 ) to getLocalSyncFlags

for i := 0; i < parallel; i++ {
s := local.NewSyncManager(ctx, client, parallel, preSignMode)
remotePath := remote.GetPath()
if recursive {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid flow also for !recursive
We should probably eliminate this param

@@ -129,18 +129,18 @@ func (s *SyncManager) apply(ctx context.Context, rootPath string, remote *uri.UR
return nil
}

func (s *SyncManager) download(ctx context.Context, rootPath string, remote *uri.URI, change *Change) error {
if err := fileutil.VerifyRelPath(strings.TrimPrefix(change.Path, uri.PathSeparator), rootPath); err != nil {
func (s *SyncManager) Download(ctx context.Context, rootPath string, remote *uri.URI, path string) error {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not we really need to externalize the Download and Upload methods

return fmt.Errorf("download %s failed: %w", change.Path, err)
}
case ChangeSourceLocal:
// we wrote something, upload it!
if err := s.upload(ctx, rootPath, remote, change); err != nil {
// we wrote something, Upload it!
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revert

@@ -98,12 +98,12 @@ func (s *SyncManager) apply(ctx context.Context, rootPath string, remote *uri.UR
switch change.Source {
case ChangeSourceRemote:
// remotely changed something, download it!
if err := s.download(ctx, rootPath, remote, change); err != nil {
if err := s.Download(ctx, rootPath, remote, change.Path); err != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏽 on change.Path

@github-actions
Copy link

github-actions bot commented Sep 28, 2023

♻️ PR Preview 0082297 has been successfully destroyed since this PR has been closed.

🤖 By surge-preview

@idanovo idanovo requested a review from N-o-Z September 28, 2023 06:19
Copy link
Member

@N-o-Z N-o-Z left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving changes pending @ozkatz response - I do believe we should change the default behavior to pre-sign=true

@idanovo idanovo merged commit aa222f4 into master Oct 1, 2023
34 checks passed
@idanovo idanovo deleted the 6321-lakectl-fs-use-sync-manager branch October 1, 2023 17:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/lakectl Issues related to lakeFS' command line interface (lakectl) include-changelog PR description should be included in next release changelog v1.0.0-blocker Issues that should be closed before going out with v1.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Modify lakectl fs download to use the SyncManager from local command
2 participants