The ARCHIVER Test Validation Suite

The ARCHIVER Test Validation Suite

To ensure that commercial cloud service offerings conform to the tender requirements and satisfy the needs of the research community, ARCHIVER  assembled a benchmarking and service validation test suite that leverage on the testing activities that took place during the HNSciCloud project. These tests derive from multiple use cases from several scientific domains and are based on scientific workload deployments in domains such as:

• Machine/Deep Learning
• Compute Benchmarking
• Cloud Storage
• Networking bandwidth availability and performance
• Multi-cloud federated hybrid cloud deployments using containers (Kubernetes)

The test suite is maintained on Github with associated documentation, distributed under the AGPL open source license. The test suite design is as modular and autonomous as possible for easy deployment, using an abstraction layer based on Docker and Kubernetes for orchestration, using Terraform for resource provisioning, and currently pushing the results to be stored in the CERN cloud. The test suite also gives the researcher the option to repatriate all associated data from the commercial cloud provider to Zenodo.

This test suite is used to validate the technical readiness level (TRL) of suppliers against the community's requirements stipulated in the ARCHIVER tender.

Check-out the ARCHIVER Test Validation Suite documentation | Access the Test Suite on Github

How does the suite work?

The process for onboarding test contributions from the research community consists of four steps:

• Researchers contact the test-suite development team in order to include a test.
• An assessment takes place in order to evaluate integration effort and requirement setup.
• Information about documentation, contact details and software licensing relevant to the test is provided.
• The test suite development team assesses the test with a software license checking tool before integrating the test.

Initial deployments of the test suite have been performed on the following cloud service providers: Exoscale (Cloudstack), Cloudscale, CloudFerro, CloudSigma, Google Cloud, Amazon Web Services, Microsoft Azure.

Started in HNSciCloud, in use in OCRE, the test suite is being further expanded in ARCHIVER.

The ARCHIVER Cloud Benchmarking Test Suite - Video presented at the ISC 2020 Digital event, the digital high performance computing conference held from 22 to 25 June 2020

Video presented at the EGI conference 2020, held from 2 to 4 November 2020, by Ignacio Peluaga from CERN