Table of contents for July/August 2021 in ADMIN Network & Security

I had an interesting discussion with my colleagues at work a few days ago that led to the question, “How do you document failure?” The discussion led us to the conclusion that failures aren’t widely documented. They should be. Think about it. When you search for a solution to a problem, you want an answer that worked for someone else. You probably don’t care about what didn’t work or the trial and error process of solving the problem – you only want to know about the successful solution. Whoever said, “Those who don’t know history are destined to repeat it,” was a genius. That person understood, intuitively perhaps, that documenting your failures is a wise choice. Applying this to system administration, or any technology job, is easy. Quite possibly, you…

In conjunction with the Cloud Native Computing Foundation and the Continuous Delivery Foundation, the Linux Foundation is now offering two new classes, focused on GitOps. These two self-paced, online training courses are designed to teach all of the skills necessary for admins to start implementing GitOps into their company's workflow. The two classes offered are as follows: • Introduction to GitOps (LFS169, https://training.linuxfoundation.org/training/introduction-to-gitops-lfs169) is a free introductory class that provides the foundation for admins to grasp the GitOps principles, tools, and best practices for the implementation of cloud-native applications running on a Kubernetes cluster. This course teaches how to set up and automate continuous delivery pipelines to a Kubernetes cluster, to increase productivity, reliability, and efficiency. • GitOps: Continuous Delivery on Kubernetes with Flux (LFS269, https://training.linuxfoundation.org/training/gitops-continuous-delivery-on-kuberneteswith-flux-lfs269/) is a course directed…

Once a private cloud with OpenStack is up and running, virtual machines can be created quickly, but you have to get there first. This process involves several steps, including installing hardware in the data center, wiring, and installing an operating system. The operating system installation is where the OpenStack component Ironic [1] enters the field. In this article, we introduce the Ironic module, which specializes in bare metal installations. Ironic first appeared as an Incubator Project in the OpenStack Icehouse release of April 2017 [2] as an offshoot of the Nova bare metal driver. It was further integrated in the course of the Juno release and finally found its place as an integrated project in the Kilo release. Bifrost Installing an operating system raises the chicken and egg problem: The…

Because of its many security weaknesses, Microsoft Windows is considered particularly vulnerable to malware such as viruses, worms, and Trojans. However, because the operating system has more or less a monopoly on the desktop, administrators in the enterprise in particular are repeatedly confronted with malware damaging Windows workstations or even rendering them unusable. In such cases, it usually makes sense to boot the damaged computers from a removable disk and then examine the mass storage devices with an external rescue system. For this purpose, many manufacturers of proprietary antivirus and anti-malware software provide free emergency CDs that use the manufacturer’s own scanners to detect and remove malware on Windows systems. In many cases, they can also be used to repair damaged system files so that the Windows system can be…

When a Kubernetes laboratory environment is required, the excellent localized Minikube and the tiny production-ready k3s distributions are fantastic. They permit a level of interaction with Kubernetes that lets developers prove workloads will act as expected in staging and production environments. However, as cloud-native technology stacks lean more on the managed service offerings of Kubernetes, courtesy of the popular cloud platforms, it has become easier to test more thoroughly on a managed Kubernetes service. These managed services include the Google Kubernetes Engine (GKE) [1], Azure Kubernetes Service (AKS) [2], and Amazon Elastic Kubernetes Service (EKS) [3]. In this article, I walk through the automated deployment of the Amazon Elastic Kubernetes Service (EKS), a mature and battle-hardened offering that is widely trusted for production workloads all over the world by household…

Holistic system security means paying attention to even the smallest detail to avoid it becoming an attack vector. One example of these small, barely noticed, but still potentially very dangerous parts of IT that repeatedly cause serious security issues is shell scripts. Employing best practices can ensure secure script programming. Letting Off Steam Impressive proof of the potential harmfulness of shell scripts was provided by US software vendor Valve. The Linux-based version of the Steam game service included a script that was normally only responsible for minor setup tasks [1]. Unfortunately, it contained the following line: This command, which is responsible for deleting the $STEAMROOT directory, gets into trouble if the environment variable is not set. Bash does not throw an error but simply “disassembles” the environment variable into an…

Time is money, and sometimes that means you need a faster way to process data. Solid state drives (SSDs) and, more specifically, non-volatile memory express (NVMe) devices have helped alleviate the burden of processing data to and from a backing store. However, at times, even SSD technology is not quite fast enough, which is where the RAM drive comes into the picture. Typically, the RAM drive is used as temporary storage for two reasons: Its capacities tend to be lower (because the technology is more expensive), and more importantly, it is a volatile technology; that is, if the system were to lose power or go into an unstable state, the contents of that RAM drive would disappear. Depending on the type of data being processed, the reward can often outweigh…

Ubuntu Pro on Google Cloud is a new platform that not only offers a 10-year maintenance commitment, but live kernel patching; officially certified components to enable operating environments under compliance regimes such as FedRAMP, HIPAA, PCI, GDPR, and ISO; certified FIPS 140-2 components; security dashboard for Security Command Center and Managed Apps; and all standard optimizations found in the regular Ubuntu releases. According to Jung Yang, VP and GM Compute at Google Cloud, “The availability of Ubuntu Pro on Google Cloud will offer our enterprise customers the additional security and compliance services needed for their mission-critical workloads.” To make this even more appealing, Ubuntu Pro on Google Cloud will only be 3-4.5 percent of your average computing costs, so the more resources you consume, the smaller percentage will go to…

System images are installed and preconfigured operating systems, including application software, for immediate use. They come in various forms. The best known is probably the virtual system, which is used in the platform as a service (PaaS) cloud. Other uses include preloading desktop systems and laptops or SD card images for single-board computers. You can create system images step by step by installing and configuring a system according to your wishes or existing standards and then creating a copy of the installation. However, don’t forget to remove some of the configuration files when you’re done (e.g., those containing the characteristics of the source system, such as MAC addresses or the hard drive IDs). Of course, this procedure is relatively time consuming and prone to error. If you are looking for…

The ownCloud Infinite Scale (oCIS) enterprise file-sharing and syncing software stack is a fast and scalable yet complete rewrite of the classic ownCloud. The current version of oCIS is the technology preview 1.8.0 (as of July 2021). Go, Vue.js, and a modular approach replace the outdated PHP, database, and POSIX filesystem the older versions needed. In internal documents, ownCloud promises up to 10 times better performance than ownCloud 10.6.0 published in December 2020 and, so far, the last PHP-based ownCloud variant. Some History A shooting star in 2010, ownCloud hasn’t make it into the news that much recently, other than with its fork Nextcloud [1]. Whoever thinks that might be attributed to Nextcloud’s media success and big community misses the point, though. Over the years, the two companies that once…

Not often are you compelled to test unfamiliar software in the time it takes to read just a few lines on its website. Of course, the software has to be extremely accessible to allow you to get started quickly. An interesting application that recently fell into this category is Kind [1], a multinode Kubernetes distribution that is quick to install and configure for testing. The reason Kind was so intriguing is that it uses Docker containers for its nodes. Think about that for a minute and consider how versatile such an approach would be for quick testing. The clever Kind manages to squeeze a Kubernetes cluster inside containers. In this article, I look at creating a freshly built Kind cluster installation to show off the infrastructure created by Kind. You…

On many networks, Active Directory (AD) is the must-have setup for authentication and assignment of rights and as a directory service. With such a central service, everything should run smoothly with PowerShell automation. In this article, I show you how to search in AD, how to secure critical accounts, and which Power- Shell helpers you will want to use. Administrators have gained a lot of experience in maintaining and operating Active Directory over its 20-year history. The tools and how they programmatically and automatically trigger changes in the directory have also changed, both in terms of data administration (i.e., managing users, computers, service accounts, and all the other objects in the directory) and in terms of the scripts for controlling the directory service itself (i.e., the service that runs on…

A topic that system administrators learn as they gain experience is called the “LD_PRELOAD Trick.” This trick can help fix misbehaving applications, upgrade applications, and even improve application performance. Of course, it is not really a trick, just the use of a feature in *nix operating systems. Have you ever installed an application on Linux and tried to run it only to be told the application can’t be found? To debug the issue, probably the first thing to check is your PATH [1], which is “an environment variable … that tells the shell which directories to search for executable files.” In short, the path tells Linux where to look for applications. If the application is not in the path, then Linux “thinks” it does not exist. Fortunately, environment variables in…

The venerable curses [1] library first enabled plotting graphics on a Unix terminal, allowing the emergence of dynamic, interactive tools such as top [2] and its brood of descendants to improve greatly the ability to monitor a system’s state continuously. The transition from static tables of numbers to charts and sometimes even dynamic data representations was followed by new implementations of terminal user interface (TUI) libraries, including Ncurses [3] and Newt [4]. Jeff Layton has previously graced these pages with a tutorial on how TUI tools are developed today [5], and yours truly followed in the next issue with a survey of network monitoring tools adopting in-terminal graphics [6]. Two years later, I revisit the subject to witness the impressive improvement of the state of the art taking place since.…

Linux Kernel 5.13 Released Linus Torvalds, the creator of Linux, has made the latest kernel available, after what was one of the smoothest development processes in recent memory. Torvalds wrote in his weekly “State of the Kernel” post: “So we had quite the calm week since rc7, and I see no reason to delay 5.13.” Torvalds continued to say, “if the last week was small and calm, 5.13 overall is actually fairly large. In fact, it's one of the bigger 5.x releases, with over 16k commits (over 17k if you count merges), from over 2k developers.” What can you expect in the 5.13 kernel? Some of the features that saw the most commits include Apple M1 support, early support for wireless wide area networks (WWANs), Microsoft's Azure Network Adapter, the…

To avoid issues that have plagued some systems, the Linux kernel will now unconditionally reserve the first 1MB of RAM. Linux developers have always known the first 64K of system memory can be easily corrupted by certain BIOSes. Another issue they've had to deal with was Intel Sandy Bridge graphics chips accessing memory below the 1MB mark. Both of these situations can lead to serious problems with system performance or even booting. Recently, however, a bug report was filed dealing with unbootable AMD Ryzen systems when the Linux kernel version 5.13 was in use. This happened after developer Linux_ Chemist discovered a system with an AMD Ryzen 7 3700x was unable to boot kernel 5.13 and discovered booting worked fine with CONFIG_X86_RESERVE_LOW=640 set. That bug was set to urgent and…

The subject of bare metal life-cycle management is a huge topic for providers today (see the “Early Efforts” box). Red Hat, Canonical, and SUSE all have powerful tools on board for this task. Third-party vendors are also trying to grab a piece of the pie, one of them being Foreman, which enjoys huge popularity. A vendor you might not expect is now also getting into the mix, with Equinix launching its Tinkerbell tool. Primarily a provider of data center and network infrastructure, Equinix is looking to manage a kind of balancing act with Tinkerbell. The tool is intended to enable customers to provision bare metal nodes in Equinix data centers just as easily as virtual instances in cloud environments. Open Source Tinkerbell With its Metal service, Equinix has been hunting…

Vembu BDR Suite [1] helps you back up Hyper-V and VMware ESXi environments with up to 10 virtual machines (VMs) free of charge. In this article, I back up a Hyper-V host and start by installing an agent for the connection – the Vembu integration service (VIS) – that is also capable of backing up server-based services such as Exchange or SQL Server on the VMs. You do not need to install any additional software on the virtual machines themselves. In addition to standalone Hyper-V hosts, you can also back up clusters with Vembu. For the backup, the vendor draws on the Hyper-V volume shadow copy service (VSS). The VMs are therefore included in the backup, instead of just copying the virtual hard drives and their data. Vembu BDR Suite…

Analyzing computer systems after a total failure (e.g., after an attack with malware) is the task of forensic specialists. With the appropriate tools, they can reconstruct log data, web history, or image data and detect so-called indicators of compromise. In this article, I introduce you to the Autopsy Sleuth Kit tool and show you how to use it for forensic analyses. After immediately provisioning alternative systems to secure business operations, one important task after a cyber incident in the enterprise is to process the incident and analyze the affected systems. In addition to countless commercial tools for the analysis and reconstruction of logs and data, you can also find very good, freely available, open source tools – such as the Sleuth Kit tool collection and its associated graphical user interface,…

Whether you are new to continuous integration/continuous deployment (CI/CD) pipelines and the world of DevOps or fully familiar with such practices, one name always comes to the fore when discussing the automation of processes: Jenkins [1]. The highly popular automation server is open source, and although capable of fulfilling many types of complex tasks, it is equally accessible to novice users through its straightforward user interface (UI). In this article, I look at how I solved a recent headache on a project in which I was running Jenkins as a Docker container for the demonstration of a vendor-supplied Jenkins plugin. Although I haven’t used Jenkins much in the past, the quandary I faced and, most importantly, resolved could affect novice and advanced users alike. The problem was that once I…

Sometimes, the easiest and quickest way to determine the root cause of a slow drive is to check its local logging data. The method by which this log data is stored will differ by the drive type, but in the end, the results are generally the same. For instance, a SCSI-based drive such as a Serial Attached SCSI (SAS) drive collects drive log data and general metrics in something called the SCSI log pages (plural because each page separates the collected data into its respective category). The easiest way to access this data is by using the sg3_utils package available for Linux. To find out what categories the drive supports, execute the sg_logs binary with the SAS drive or SCSI generic identifier in which you are interested (Listing 1). As…