Skip to content

Commit

Permalink
Add delete and list generation operations
Browse files Browse the repository at this point in the history
  • Loading branch information
@svanderburg
svanderburg committed Feb 13, 2018
1 parent a0342ed commit 3ebfcee
Show file tree
Hide file tree
Showing 4 changed files with 182 additions and 28 deletions.
2 changes: 2 additions & 0 deletions NEWS
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ Version 0.8

- The copy closure command now partitions the requisites to prevent hitting the "arguments too long" error

- The disnix-env command now also takes the --list-generations option to display coordinator profile generations and --delete-generations, --delete-all-generations to delete them

Version 0.7
===========
- When moving state it will no longer be deleted by default. Instead, the user must provide --delete-state
Expand Down
42 changes: 42 additions & 0 deletions doc/manual/basicusage.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1092,6 +1092,15 @@ $ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix
on the coordinator machine, then the services and their intra-dependencies are transferred to the target machines
in the network. Finally, the services are activated.
</para>

<para>
When running the following command-line instruction:
<screen>
$ disnix-env --list-generations
1 2018-02-13 13:22:35 (current)
</screen>
We can see an overview of deployment generations. Because this is our initial deployment, we only have one generation.
</para>
</section>

<section>
Expand All @@ -1118,6 +1127,17 @@ $ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix
from the coordinator machine to <code>test2</code> and redeploys the web application front-end to connect to the new location of the
<code>ZipcodeService</code>. All the other parts of the system will remain untouched.
</para>

<para>
When running the following command-line instruction:
<screen>
1 2018-02-13 13:22:35
2 2018-02-13 13:30:24 (current)
</screen>
We will see that we have two configurations deployed, the current after the upgrade (generation 2) and the previous configuration (generation 1).
Because Disnix builts on top of Nix, older configurations are never removed by default. We can use the older
configuration to easily roll back to a previous configuration, when desired.
</para>
</section>

<section>
Expand Down Expand Up @@ -1172,6 +1192,28 @@ $ disnix-collect-garbage infrastructure.nix
<option>-d</option> option can be used:
<screen>
$ disnix-collect-garbage -d infrastructure.nix
</screen>
</para>
<para>
In addition to the target machines, the coordinator machine also keeps packages of all configurations
unless they are declared obsolete and garbage collected. We can use the following command
to delete all older generations on the coordinator machine:
<screen>
$ disnix-env --delete-generations old
</screen>
</para>
<para>
When running the garbage collector on the coordinator machine, all obsolete packages of the obsolete
configurations will be removed:
<screen>
$ nix-collect-garbage -d
</screen>
</para>
<para>
It also possible to discard all profile generations. This is useful, for example, when the
target machines are no longer available:
<screen>
$ disnix-env --delete-all-generations
</screen>
</para>
</section>
Expand Down
113 changes: 89 additions & 24 deletions scripts/disnix-env.in
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,9 @@ showUsage()
Usage: $0 -s services_nix -i infrastructure_nix -d distribution_nix [OPTION]
or: $0 --rollback [OPTION]
or: $0 --switch-generation NUM [OPTION]
or: $0 --list-generations [OPTION]
or: $0 --delete-generations NUM [OPTION]
or: $0 --delete-all-generations NUM [OPTION]
The command \`disnix-env' is used to install, upgrade, or roll back a
service-oriented system in a given environment.
Expand Down Expand Up @@ -75,6 +78,13 @@ Options:
--switch-to-generation=NUM Switches to a specific profile generation
--rollback Switches back to the previously deployed
configuration
--list-generations Lists all profile generations of the current
deployment
--delete-generations=NUM Deletes the specified generations. The number
can correspond to generation numbers, days
(d postfix) or 'old'.
--delete-all-generations Deletes all profile generations. This is
useful when a deployment has been discarded
--target-property=PROP The target property of an infrastructure
model, that specifies how to connect to the
remote Disnix interface. (Defaults to
Expand Down Expand Up @@ -151,7 +161,7 @@ EOF

# Parse valid argument options

PARAMS=`@getopt@ -n $0 -o s:i:d:p:m:hv -l services:,infrastructure:,distribution:,rollback,switch-to-generation:,interface:,target-property:,deploy-state,profile:,max-concurrent-transfers:,build-on-targets,coordinator-profile-path:,no-upgrade,no-lock,no-coordinator-profile,no-target-profiles,no-migration,delete-state,depth-first,keep:,show-trace,help,version -- "$@"`
PARAMS=`@getopt@ -n $0 -o s:i:d:p:m:hv -l services:,infrastructure:,distribution:,rollback,switch-to-generation:,list-generations,delete-generations:,delete-all-generations,interface:,target-property:,deploy-state,profile:,max-concurrent-transfers:,build-on-targets,coordinator-profile-path:,no-upgrade,no-lock,no-coordinator-profile,no-target-profiles,no-migration,delete-state,depth-first,keep:,show-trace,help,version -- "$@"`

if [ $? != 0 ]
then
Expand All @@ -176,12 +186,22 @@ do
distributionFile=`readlink -f $2`
;;
--rollback)
switchGeneration=1
operation="switchGenerations"
;;
--switch-to-generation)
switchGeneration=1
operation="switchGenerations"
generationId=$2
;;
--list-generations)
operation="listGenerations"
;;
--delete-generations)
operation="deleteGenerations"
generations="$2"
;;
--delete-all-generations)
operation="deleteAllGenerations"
;;
-m|--max-concurrent-transfers)
maxConcurrentTransfersArg="-m $2"
;;
Expand Down Expand Up @@ -241,7 +261,7 @@ do
exit 0
;;
esac

shift
done

Expand All @@ -255,13 +275,6 @@ source @datadir@/@PACKAGE@/checks

# Validate the given options

if [ "$switchGeneration" != "1" ]
then
checkServicesFile
checkInfrastructureFile
checkDistributionFile
fi

checkClientInterface
checkTargetProperty
checkProfile
Expand All @@ -279,10 +292,17 @@ else
profileDir="$coordinatorProfilePath"
fi

# Execute operations
# Operation functions

if [ "$switchGeneration" = "1" ]
then
checkModels()
{
checkServicesFile
checkInfrastructureFile
checkDistributionFile
}

switchGenerations()
{
# If no generation link has been requested, figure out what the previous one is
if [ "$generationId" = "" ]
then
Expand All @@ -296,7 +316,7 @@ then
else
generation="$profile-$generationId-link"
fi

if [ -e "$profileDir/$generation" ]
then
# Retrieve the manifest of the requested profile generation
Expand All @@ -305,21 +325,66 @@ then
echo "[coordinator]: Cannot switch profile generations, because the requested profile does not exist!" >&2
exit 1
fi
else
}

listGenerations()
{
nix-env -p $profileDir/$profile --list-generations
}

deleteGenerations()
{
nix-env -p $profileDir/$profile --delete-generations "$generations"
}

deleteAllGenerations()
{
rm -f $profileDir/$profile
rm -f $profileDir/$profile-*-link
}

produceManifest()
{
if [ "$buildOnTargets" = "1" ]
then
disnix-delegate -s $servicesFile -i $infrastructureFile -d $distributionFile --target-property $targetProperty --interface $interface $maxConcurrentTransfersArg $showTraceArg
fi

echo "[coordinator]: Building manifest..."
manifest=`disnix-manifest -s $servicesFile -i $infrastructureFile -d $distributionFile --target-property $targetProperty --no-out-link --interface $interface $deployStateArg $showTraceArg`
fi
}

# Determine the name of the previous manifest file
if [ -d "$profileDir" ] && [ "$(echo $profileDir/$profile-*-link)" != "" ]
then
oldManifestArg="-o $(ls $profileDir/$profile-*-link | sort -V | tail -1)"
fi
deploy()
{
# Determine the name of the previous manifest file
if [ -d "$profileDir" ] && [ "$(echo $profileDir/$profile-*-link)" != "" ]
then
oldManifestArg="-o $(ls $profileDir/$profile-*-link | sort -V | tail -1)"
fi

# Deploy the (pre)built Disnix configuration (implying a manifest file)
disnix-deploy $maxConcurrentTransfersArg $noLockArg $profileArg $noUpgradeArg $deleteStateArg $noCoordinatorProfileArg $coordinatorProfilePathArg $noTargetProfilesArg $noMigrationArg $oldManifestArg $depthFirstArg $keepArg $manifest
}

# Execute operations

# Deploy the (pre)built Disnix configuration (implying a manifest file)
disnix-deploy $maxConcurrentTransfersArg $noLockArg $profileArg $noUpgradeArg $deleteStateArg $noCoordinatorProfileArg $coordinatorProfilePathArg $noTargetProfilesArg $noMigrationArg $oldManifestArg $depthFirstArg $keepArg $manifest
case "$operation" in
switchGenerations)
switchGenerations
deploy
;;
listGenerations)
listGenerations
;;
deleteGenerations)
deleteGenerations
;;
deleteAllGenerations)
deleteAllGenerations
;;
*)
checkModels
produceManifest
deploy
;;
esac
53 changes: 49 additions & 4 deletions tests/deployment.nix
View file
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,15 @@ with import "${nixpkgs}/nixos/lib/testing.nix" { system = builtins.currentSystem
# This test should succeed.
$coordinator->mustSucceed("${env} disnix-env -s ${manifestTests}/services-complete.nix -i ${manifestTests}/infrastructure.nix -d ${manifestTests}/distribution-simple.nix");
# Check if we have one profile generation.
my $result = $coordinator->mustSucceed("${env} disnix-env --list-generations | wc -l");
if($result == 1) {
print("We have 1 profile generation\n");
} else {
die "We should have 1 profile generation, instead we have: $result";
}
# Do another rollback. Since there is no previous deployment, it should fail.
$coordinator->mustFail("${env} disnix-env --rollback");
Expand Down Expand Up @@ -89,7 +98,7 @@ with import "${nixpkgs}/nixos/lib/testing.nix" { system = builtins.currentSystem
# folder and one generation link in the target profiles folder on each
# machine.
my $result = $coordinator->mustSucceed("ls /nix/var/nix/profiles/per-user/root/disnix-coordinator | wc -l");
$result = $coordinator->mustSucceed("ls /nix/var/nix/profiles/per-user/root/disnix-coordinator | wc -l");
if($result == 2) {
print "We have only one generation symlink on the coordinator!\n";
Expand Down Expand Up @@ -119,6 +128,14 @@ with import "${nixpkgs}/nixos/lib/testing.nix" { system = builtins.currentSystem
$coordinator->mustSucceed("${env} disnix-env -s ${manifestTests}/services-complete.nix -i ${manifestTests}/infrastructure.nix -d ${manifestTests}/distribution-simple.nix");
$result = $coordinator->mustSucceed("${env} disnix-env --list-generations | wc -l");
if($result == 1) {
print "We have one generation symlink\n";
} else {
die "We should have one generation symlink, instead we have: $result"
}
$result = $coordinator->mustSucceed("ls /nix/var/nix/profiles/per-user/root/disnix-coordinator | wc -l");
if($result == 2) {
Expand Down Expand Up @@ -147,6 +164,14 @@ with import "${nixpkgs}/nixos/lib/testing.nix" { system = builtins.currentSystem
# This test should succeed.
$coordinator->mustSucceed("${env} disnix-env -s ${manifestTests}/services-complete.nix -i ${manifestTests}/infrastructure.nix -d ${manifestTests}/distribution-reverse.nix");
$result = $coordinator->mustSucceed("${env} disnix-env --list-generations | wc -l");
if($result == 2) {
print "We have two generation symlinks";
} else {
die "We should have two generation symlinks, instead we have: $result"
}
@lines = split('\n', $coordinator->mustSucceed("${env} disnix-query ${manifestTests}/infrastructure.nix"));
if($lines[3] =~ /\-testService1/) {
Expand Down Expand Up @@ -452,6 +477,19 @@ with import "${nixpkgs}/nixos/lib/testing.nix" { system = builtins.currentSystem
die "disnix-query output line 9 does not contain testService3!\n";
}
# Remove old generation test. We remove one profile generation and we
# check if it has been successfully removed
my $oldNumOfGenerations = $coordinator->mustSucceed("${env} disnix-env --list-generations | wc -l");
$coordinator->mustSucceed("${env} disnix-env --delete-generations 1");
my $newNumOfGenerations = $coordinator->mustSucceed("${env} disnix-env --list-generations | wc -l");
if($newNumOfGenerations == $oldNumOfGenerations - 1) {
print "We have successfully removed one generation!\n";
} else {
die "We seem to have removed more than one generation, new: $newNumOfGenerations, old: $oldNumOfGenerations";
}
# Test disnix-capture-infra. Capture the container properties of all
# machines and generate an infrastructure expression from it. It should
# contain: "foo" = "bar"; twice.
Expand All @@ -475,9 +513,9 @@ with import "${nixpkgs}/nixos/lib/testing.nix" { system = builtins.currentSystem
# Test disnix-reconstruct. Because nothing has changed the coordinator
# profile should remain identical.
my $oldNumOfGenerations = $coordinator->mustSucceed("ls /nix/var/nix/profiles/per-user/root/disnix-coordinator | wc -l");
$oldNumOfGenerations = $coordinator->mustSucceed("ls /nix/var/nix/profiles/per-user/root/disnix-coordinator | wc -l");
$coordinator->mustSucceed("${env} disnix-reconstruct ${manifestTests}/infrastructure.nix");
my $newNumOfGenerations = $coordinator->mustSucceed("ls /nix/var/nix/profiles/per-user/root/disnix-coordinator | wc -l");
$newNumOfGenerations = $coordinator->mustSucceed("ls /nix/var/nix/profiles/per-user/root/disnix-coordinator | wc -l");
if($oldNumOfGenerations == $newNumOfGenerations) {
print "The amount of manifest generations remained the same!\n";
Expand All @@ -488,7 +526,14 @@ with import "${nixpkgs}/nixos/lib/testing.nix" { system = builtins.currentSystem
# Test disnix-reconstruct. First, we remove the old manifests. They
# should have been reconstructed.
$coordinator->mustSucceed("rm /nix/var/nix/profiles/per-user/root/disnix-coordinator/*");
$result = $coordinator->mustSucceed("${env} disnix-env --delete-all-generations | wc -l");
if($result == 0) {
print "We have no symlinks";
} else {
die "We should have no symlinks, instead we have: $result";
}
$coordinator->mustSucceed("${env} disnix-reconstruct ${manifestTests}/infrastructure.nix");
$result = $coordinator->mustSucceed("ls /nix/var/nix/profiles/per-user/root/disnix-coordinator | wc -l");
Expand Down

0 comments on commit 3ebfcee

Please sign in to comment.