Github News

• 

  GitHub Availability Report: April 2023
  • sorangutan  

  1
  1
  Posts
  4
  Views

  

  In April, we experienced four incidents that resulted in degraded performance across GitHub services. This report also sheds light into three March incidents that resulted in degraded performance across GitHub services.

  March 27 12:25 UTC (lasting 1 hour and 33 minutes)

  On March 27 at 12:14 UTC, users began to see degraded experience with Git Operations, GitHub Issues, pull requests, GitHub Actions, GitHub API requests, GitHub Codespaces, and GitHub Pages. We publicly statused Git Operations 11 minutes later, initially to yellow, followed by red for other impacted services. Full functionality was restored at 13:17 UTC.

  The cause was traced to a change in a frequently-used database query. The query alteration was part of a larger infrastructure change that had been rolled out gradually, starting in October 2022, then more quickly beginning February 2023, completing on March 20, 2023. The change increased the chance of lock contention, leading to increased query times and eventual resource exhaustion during brief load spikes, which caused the database to crash. An initial automatic failover solved this seamlessly, but the slow query continued to cause lock contention and resource exhaustion, leading to a second failover that did not complete. Mitigation took longer than usual because manual intervention was required to fully recover.

  The query causing lock tension was disabled via feature flag, and then refactored. We have added additional monitoring of relevant database resources so as not to reach resource exhaustion, and detect similar issues earlier in our staged rollout process. Additionally, we have enhanced our query evaluation procedures related to database lock contention, along with improved documentation and training material.

  March 29 14:21 UTC (lasting 4 hour and 57 minutes)

  On March 29 at 14:10 UTC, users began to see a degraded experience with GitHub Actions with their workflows not progressing. Engineers initially statused GitHub Actions nine minutes later. GitHub Actions started recovering between 14:57 UTC and 16:47 UTC before degrading again. GitHub Actions fully recovered the queue of backlogged workflows at 19:03 UTC.

  We determined the cause of the impact to be a degraded database cluster. Contributing factors included a new load source from a background job querying that database cluster, maxed out database transaction pools, and underprovisioning of vtgate proxy instances that are responsible for query routing, load balancing, and sharding. The incident was mitigated through throttling of job processing and adding capacity, including overprovisioning to speed processing of the backlogged jobs.

  After the incident, we identified that the pool, found_rows_pool managed by the vtgate layer, was overwhelmed and unresponsive. This pool became flooded and stuck due to contention between inserting data into and reading data from the tables in the database. This contention led to us being unable to progress any new queries across our database cluster.

  The health of our database clusters is a top priority for us and we have taken steps to reduce contentious queries on our cluster over the last few weeks. We also have taken multiple actions from what we learned in this incident to improve our telemetry and alerting to allow us to identify and act on blocking queries faster. We are carefully monitoring the cluster health and are taking a close look into each component to identify any additional repair items or adjustments we can make to improve long-term stability.

  March 31 01:07 UTC (lasting 2 hours)

  On March 31 at 00:06 UTC, a small percentage of users started to receive consistent 500 error responses on pull request files pages. At 01:07 UTC, the support team escalated reports from customers to engineering who identified the cause and statused yellow nine minutes later. The fix was deployed to all production hosts by 02:07 UTC.

  We determined the source of the bug to be a notification to promote a new feature. Only repository admins who had not enabled the new feature or dismissed the notification were impacted. An expiry date in the configuration of this notification was set incorrectly, which caused a constant that was still referenced in code to no longer be available.

  We have taken steps to avoid similar issues in the future by auditing the expiry dates of existing notices, preventing future invalid configurations, and improving test coverage.

  April 18 09:28 UTC (lasting 11 minutes)

  On April 18 at 09:22 UTC, users accessing any issues or pull request related entities experienced consistent 5xx responses. Engineers publicly statused pull requests to red and issues six minutes later. At 09:33 UTC, the root cause self-healed and traffic recovered. The impact resulted in an 11 minute outage of access to issues and pull request related artifacts. After fully validating traffic recovery, we statused green for issues at 09:42 UTC.

  The root cause of this incident was a planned change in our database infrastructure to minimize the impact of unsuccessful deployments. As part of the progressive rollout of this change, we deleted nodes that were taking live traffic. When these nodes were deleted, there was an 11 minute window where requests to this database cluster failed. The incident was resolved when traffic automatically switched back to the existing nodes.

  This planned rollout was a rare event. In order to avoid similar incidents, we have taken steps to review and improve our change management process. We are updating our monitoring and observability guidelines to check for traffic patterns prior to disruptive actions. Furthermore, we’re adding additional review steps for disruptive actions. We have also implemented a new checklist for change management for these types of infrequent administrative changes that will prompt the primary operator to document the change and associated risks along with mitigation strategies.

  April 26 23:26 UTC (lasting 1 hour and 04 minutes)

  On April 26 at 23:26 UTC, we were notified of an outage with GitHub Copilot. We resolved the incident at 00:29 UTC.

  Due to the recency of this incident, we are still investigating the contributing factors and will provide a more detailed update in next month’s report.

  April 27 08:59 UTC (lasting 57 minutes)

  On April 26 at 08:59 UTC, we were notified of an outage with GitHub Packages. We resolved the incident at 09:56 UTC.

  Due to the recency of this incident, we are still investigating the contributing factors and will provide a more detailed update in next month’s report.

  April 28 12:26 UTC (lasting 19 minutes)

  On April 28 at 12:26 UTC, we were notified of degraded availability for GitHub Codespaces. We resolved the incident at 12:45 UTC.

  Due to the recency of this incident, we are still investigating the contributing factors and will provide a more detailed update in next month’s report.

  ---

  Please follow our status page for real-time updates on status changes. To learn more about what we’re working on, check out the GitHub Engineering Blog.

  
  
  https://github.blog/2023-05-03-github-availability-report-april-2023/
• 

  CLI tricks every developer should know
  • sorangutan  

  1
  1
  Posts
  9
  Views

  

  The CLI is a critical component of a developer’s toolkit—it’s a trusty sidekick that offers flexibility and control. You can tell it what to do by simply typing in a specific command, and it will execute those commands like moving files, running programs, or even starting up a server, immediately. That being said, the CLI can be daunting to beginners, especially when you’re not sure which commands to run (and who hasn’t turned to Google to find a command they need?).

  In this blog, we’ve compiled some important tricks and commands that every developer should know from GitHub’s own engineers. By mastering these basic techniques, developers can become more efficient at working with the command line and gain a deeper understanding of how the underlying operating system and programs work.

  Components of the CLI

  The CLI has two main components: the shell and the command. The shell is the interface that allows the user to enter commands and commands are the instructions that tell the computer what to do. Shells also provide a way to customize and extend the behavior of the CLI. With a shell, users can create their own scripts and aliases to automate tasks or simplify complex commands, and they can customize the behavior of the shell itself using configuration files. For this blog post, all of the examples are for Bash since it’s the most widely used shell. And if you’re using Windows, Windows Subsystem for Linux (WSL) is available if you’d like to use a Bash terminal.

  To learn more about shells, you can check out our shell documentation here.

  Keyboard shortcuts in the CLI

  One of the easiest ways to improve your productivity on the command line is to learn some keyboard shortcuts. Below, you’ll find popular shortcuts that can save you time and effort on navigating and executing demands.

  CTRL + C: Cancel the current command

  CTRL + A: Move the cursor to the beginning of the line

  CTRL + E: Move the cursor to the end of the line

  CTRL + L: Clear the terminal screen

  CTRL + _: Undo the last edit

  CTRL-D: Exit shell session

  TAB:Auto-complete commands, file names, and paths

  CLI command history shortcuts

  The command history allows you to quickly access and reuse previously executed commands rather than retype or search for the whole command. Here are a few to try out for yourself:

  history n: Type this in the terminal to access the history

  !!: Execute the last command

  CTRL + R: Reverse search for a previously executed command

  CTRL + P: Move to the previous command in the history

  CTRL + N: Move to the next command in the history

  CTRL + G: Abort search

  Perform operations on multiple files with wildcards

  Wildcards are characters that take the place of one or more other characters, and they’re used to increase the efficiency and flexibility of searches. They come in handy when you want to perform an operation on multiple files or directories that have similar names or patterns, and can save you a lot of time and effort by allowing you to specify patterns rather than list each individual file. There are three types of command wildcards:

  ?: Matches a single character. For example, if you type d?g, it will match anything that begins with a “d” and ends with an “g.”

  *: Matches any number of characters. If you search s*n, it will match anything between “s” and “n” no matter how many characters are between the first and last letter.

  []: Matches only the characters enclosed within the square brackets.

  Combine commands with pipes

  A pipe (represented by the symbol |) connects the output of the command on the left side of the symbol to the input of the command on the right side. This allows users to create more complex and powerful commands by combining simpler commands together.

  Here’s an example of how it’s used in a command:

  ls -la | *grep test | sort | uniq | wc -l: The final output of this command would be the number of files and directories in the current directory whose name contains the string “test.”

  When you pipe all of these commands together, you are essentially:

  • Listing all files and directories in the current directory in long format
  • Searching for all lines that contain the string “test”
  • Sorting those lines alphabetically
  • Removing any duplicate lines
  • Counting the number of remaining lines

  Note: grep is a useful CLI tool to search for matching patterns in a file. We’ll explore some more helpful CLI tools later on in this article.

  Command substitution

  Command substitution is a feature that allows you to execute a command with a different command. This helps you create more complex and dynamic commands by using the output of one command as an argument for another.

  There are two different syntaxes for command substitution:

  $(command) or command

  Here’s a simple example:

  $ echo "the date is $(date)": This will display the current date and time in a sentence.

  

  Learning the CLI commands and tricks that work for you takes time and practice. But you’ll soon find that using the command line becomes second nature, and you’ll be able to accomplish more complex tasks with ease.

  Now that we’ve covered some basics, you can begin to experiment with different options and parameters for your commands to see what they do. And stay curious! There are tons of great resources out there, like tldr or Explainshell, to help you learn new commands or shortcuts. And speaking of resources, let’s take a look at some helpful CLI tools to help you optimize the command line—and have some fun with it.

  CLI tools to know

  Command line tools are scripts, programs, or libraries created to solve problems that developers might encounter while coding. While these tools are largely focused on improving productivity, there are also CLI tools that can entertain you while you’re in the terminal (check out football-cli or spotify-tui for reference).

  Here are a few CLI tools that we recommend.

  grep or ack grep is a command-line utility that allows users to search for specific patterns in text files. Users can search for a specific word, phrase, or regular expression, and output the matching lines. ack is designed to be a faster alternative to the traditional grep command. Some prefer ack over grep because it can automatically exclude files that are not typically used for programming, such as binary files or backup files.	jq jq is a lightweight and powerful command-line JSON processor that allows developers to parse, filter, and transform JSON data using a range of query and manipulation functions. With jq, developers can easily extract specific data from large JSON files, format JSON output, and even convert between JSON and other data formats.	ImageMagick Though ImageMagick has been around since the ‘90s, this tool remains the gold standard for all things image-related. It supports a wide range of image formats and can help simplify resizing, cropping, rotating, and applying filters and effects. One of the most powerful features of ImageMagick is its ability to automate batch processing of images, which is particularly useful for web developers and designers who need to resize or convert large numbers of images at once.

  howdoi howdoi provides answers to your programming questions directly in the terminal. This command-line tool is particularly useful since you don’t have to leave the CLI or open your web browser to get immediate access to programming knowledge.	Taskwarrior Taskwarrior manages your to-do list right from the command line. It also includes a variety of reports and filters that allow you to view your tasks in different ways, like by project, due date, or priority.	GitHub CLI GitHub CLI provides a streamlined and efficient way to manage GitHub projects and workflows directly from the command line. You can perform a range of tasks, like managing pull requests, viewing and responding to issues and comments, and reviewing code changes. You can also use it to access GitHub Actions and other features on GitHub without exiting your terminal.

  The bottom line: ​​The CLI is a powerful tool for developers—especially if you prefer the keyboard over GUI interfaces. Learning how to use it and incorporating it into your daily workflow will help you become more productive and efficient, and getting familiar with these tips and tricks is a great way to start your CLI journey.

  Additional resources for CLI

  As you move further along in your developing journey, you’ll encounter different projects and pain points, which means that the commands and CLI tools you find useful might change and shift. There are so many diverse and useful CLI applications out there, and the repository awesome-cli-apps lists a bunch of excellent tools that are used and loved by developers around the world.

  What’s next for CLI

  We’ll be launching a technical preview of GitHub Copilot for CLI, which translates natural language prompts into terminal commands to help you find the exact command you need for the task at hand. Click here to get on the waitlist and try it out for yourself.

  
  
  https://github.blog/2023-04-26-cli-tricks-every-developer-should-know/
• 

  3 benefits of migrating and consolidating your source code
  • sorangutan  

  1
  1
  Posts
  24
  Views

  

  In a previous blog on consolidating your toolkit, we shared strategies to help you simplify your tech stack, which ultimately helps developers be more productive. In fact, developers are almost 60% more likely to feel equipped to do their job when they can easily find what they need.

  But there are other benefits of consolidating and simplifying your toolkit that may be surprising–especially when migrating your source code and collaboration history to GitHub.

  Today, we’ll explore three benefits that will support enterprises in a business climate where everyone is being asked to do more with less, as well as some resources to help get started on the migration journey.

  1. Enable developer self-service culture

  Some of the benefits enterprises can achieve with DevOps are improved productivity, security, and collaboration. Silos should be broken down and traditionally separated teams should be working in a cohesive and cloud native way.

  Another benefit that DevOps enables, which is a key part of the Platform Engineering technology approach, is the ability for development teams to self-service workflows, processes, and controls which traditionally have either been manual, or tightly-coupled with other teams. A great example of this was covered in a previous blog where we described how to build consistent and shared IaC workflows. IaC workflows can be created by operations teams, if your enterprise separation of duties governance policies require this, but self-serviced when needed by development teams.

  But this type of consistent, managed, and governable, self-service culture would not be possible if you have multiple source code management tools in your enterprise. If development teams have to spend time figuring out which tool has the source of truth for the workflow they need to execute, the benefits of DevOps and Platform Engineering quickly deteriorate.

  There is no better place to migrate the core of your self-service culture to than GitHub–which is the home to 100 million developers and counting. Your source code management tool should be an enabler for developer productivity and happiness or else they will be reluctant to use it. And if they don’t use it, you won’t have a self-service culture within your enterprise.

  2. Save time and money during audits

  The latest Forrester report on the economic impact of GitHub Enterprise Cloud and GitHub Advanced Security, determined a 75% improvement in time spent managing tools and code infrastructure. But one of the potentially surprising benefits is related to implementing DevOps and cloud native processes that would both help developers and auditors save time and money.

  If your tech stack includes multiple source code tools, and other development tools which may not be integrated our have overlapping capabilities, each time your security, compliance, and audit teams need to review the source of truth for your delivery artifacts, you will need to gather artifacts and setup walkthroughs for each of the tools. This can lead to days and even weeks of lost time and money on simply preparing and executing audits–taking your delivery teams away from creating business value.

  Working with GitHub customers, Forrester identified and quantified key benefits of investing in GitHub Enterprise Cloud and GitHub Advanced Security. The corresponding GitHub Ent ROI Estimate Calculator includes factors for time saving on IT Audit preparations related to the number of non-development security or audit staff involved in software development. This itself can lead to hundreds of thousands if not millions of dollars of time savings.

  What is not factored into the calculator is the potential time savings for development teams who have a single source of truth for their code and collaboration history. A simplified and centrally auditable tech stack with a single developer-friendly core source code management platform will enable consistent productivity even during traditionally time-consuming audit and compliance reviews–for both developers and non-developers.

  3. Keep up with innovation

  If you are using another source code platform besides GitHub, or if GitHub is one of several tools that are providing the overlapping functionality, some of your teams may be missing out on the amazing innovations that have been happening lately.

  Generative AI is enabling some amazing capabilities and GitHub is at the forefront with our AI pair-programmer, GitHub Copilot. The improvements to developer productivity are truly amazing and continue to improve.

  A graphic showing how many developers and companies have already used GitHub Copilot and how it’s helping improve productivity and happiness.

  GitHub continues to innovate with the news about GitHub Copilot X, which is not only adopting OpenAI’s new GPT-4 model, but introducing chat and voice for GitHub Copilot, and bringing GitHub Copilot to pull requests, the command line, and docs to answer questions on your projects.

  Innovations like this need to be rolled-out in a controlled and governable manner within many enterprises. But if your techstack is overly complex and you have several source code management tools, the roll-out may take a long time or may be stalled while security and compliance reviews take place.

  However, if your development core is GitHub, security and compliance reviews can happen once, on a centrally managed platform that is well understood and secure. And you’ll be front row for all of the amazing new innovations that GitHub will be releasing down the road.

  Get started today

  If you are planning on migrating your source code and collaboration history to GitHub and have questions, thankfully, many other enterprises have done this already with great success and there are resources to help you through the process.

  Visit our GitHub Enterprise Importer documentation for details on supported migration paths, guides for different migration sources, and more.

  If you want to learn more about how GitHub can benefit your business, while increasing developer velocity and collaboration, see how GitHub Enterprise can help.

  
  
  https://github.blog/2023-04-14-3-benefits-of-migrating-and-consolidating-your-source-code/
• 

  How generative AI is changing the way developers work
  • sorangutan  

  1
  1
  Posts
  21
  Views

  

  During a time when computers were solely used for computation, the engineer, Douglas Engelbart, gave the “mother of all demos,” where he reframed the computer as a collaboration tool capable of solving humanity’s most complex problems. At the start of his demo, he asked audience members how much value they would derive from a computer that could instantly respond to their actions.

  You can ask the same question of generative AI models. If you had a highly responsive generative AI coding tool to brainstorm new ideas, break big ideas into smaller tasks, and suggest new solutions to problems, how much more creative and productive could you be?

  This isn’t a hypothetical question. AI-assisted engineering workflows are quickly emerging with new generative AI coding tools that offer code suggestions and entire functions in response to natural language prompts and existing code. These tools, and what they can help developers accomplish, are changing fast. That makes it important for every developer to understand what’s happening now—and the implications for how software is and will be built.

  In this article, we’ll give a rundown of what generative AI in software development looks like today by exploring:

  • The unique value generative AI brings to the developer workflow
  • How generative AI coding tools are designed and built
  • Why developers should care about large language models
  • How developers are using generative AI coding tools

  The unique value generative AI brings to the developer workflow

  AI and automation have been a part of the developer workflow for some time now. From machine learning-powered security checks to CI/CD pipelines, developers already use a variety of automation and AI tools, like CodeQL on GitHub, for example.

  While there’s overlap between all of these categories, here’s what makes generative AI distinct from automation and other AI coding tools:

  Automation: You know what needs to be done, and you know of a reliable way to get there every time.	Rules-based logic: You know the end goal, but there’s more than one way to achieve it.	Machine learning: You know the end goal, but the amount of ways to achieve it scales exponentially.	Generative AI: You have big coding dreams, and want the freedom to bring them to life.
  You want to make sure that any new code pushed to your repository follows formatting specifications before it’s merged to the main branch. Instead of manually validating the code, you use a CI/CD tool like GitHub Actions to trigger an automated workflow on the event of your choosing (like a commit or pull request).	You know some patterns of SQL injections, but it’s time consuming to manually scan for them in your code. A tool like Code QL uses a system of rules to sort through your code and find those patterns, so you don’t have to do it by hand.	You want to stay on top of security vulnerabilities, but the list of SQL injections continues to grow. A coding tool that uses a machine learning (ML) model, like Code QL, is trained to not only detect known injections, but also patterns similar to those injections in data it hasn’t seen before. This can help you increase recognition of confirmed vulnerabilities and predict new ones.	Generative AI coding tools leverage ML to generate novel answers and predict coding sequences. A tool like GitHub Copilot can reduce the amount of times you switch out of your IDE to look up boilerplate code or help you brainstorm coding solutions. Shifting your role from rote writing to strategic decision making, generative AI can help you reflect on your code at a higher, more abstract level—so you can focus more on what you want to build and spend less time worrying about how.

  How are generative AI coding tools designed and built?

  Building a generative AI coding tool requires training AI models on large amounts of code across programming languages via deep learning. (Deep learning is a way to train computers to process data like we do—by recognizing patterns, making connections, and drawing inferences with limited guidance.)

  To emulate the way humans learn patterns, these AI models use vast networks of nodes, which process and weigh input data, and are designed to function like neurons. Once trained on large amounts of data and able to produce useful code, they’re built into tools and applications. The models can then be plugged into coding editors and IDEs where they respond to natural language prompts or code to suggest new code, functions, and phrases.

  Before we talk about how generative AI coding tools are made, let’s define what they are first. It starts with LLMs, or large language models, which are sets of algorithms trained on large amounts of code and human language. Like we mentioned above, they can predict coding sequences and generate novel content using existing code or natural language prompts.

  Today’s state-of-the-art LLMs are transformers. That means they use something called an attention mechanism to make flexible connections between different tokens in a user’s input and the output that the model has already generated. This allows them to provide responses that are more contextually relevant than previous AI models because they’re good at connecting the dots and big-picture thinking.

  Here’s an example of how a transformer works. Let’s say you encounter the word log in your code. The transformer node at that place would use the attention mechanism to contextually predict what kind of log would come next in the sequence.

  Let’s say, in the example below, you input the statement from math import log. A generative AI model would then infer you mean a logarithmic function.

  

  And if you add the prompt from logging import log, it would infer that you’re using a logging function.

  

  Though sometimes a log is just a log.

  

  LLMs can be built using frameworks besides transformers. But LLMs using frameworks, like a recurrent neural network or long short-term memory, struggle with processing long sentences and paragraphs. They also typically require training on labeled data (making training a labor-intensive process). This limits the complexity and relevance of their outputs, and the data they can learn from.

  Transformer LLMs, on the other hand, can train themselves on unlabeled data. Once they’re given basic learning objectives, LLMs take a part of the new input data and use it to practice their learning goals. Once they’ve achieved these goals on that portion of the input, they apply what they’ve learned to understand the rest of the input. This self-supervised learning process is what allows transformer LLMs to analyze massive amounts of unlabeled data—and the larger the dataset an LLM is trained on, the more they scale by processing that data.

  Why should developers care about transformers and LLMs?

  LLMs like OpenAI’s GPT-3, GPT-4, and Codex models are trained on an enormous amount of natural language data and publicly available source code. This is part of the reason why tools like ChatGPT and GitHub Copilot, which are built on these models, can produce contextually accurate outputs.

  Here’s how GitHub Copilot produces coding suggestions:

  • All of the code you’ve written so far, or the code that comes before the cursor in an IDE, is fed to a series of algorithms that decide what parts of the code will be processed by GitHub Copilot.
  • Since it’s powered by a transformer-based LLM, GitHub Copilot will apply the patterns it’s abstracted from training data and apply those patterns to your input code.
  • The result: contextually relevant, original coding suggestions. GitHub Copilot will even filter out known security vulnerabilities, vulnerable code patterns, and code that matches other projects.

  Keep in mind: creating new content such as text, code, and images is at the heart of generative AI. LLMs are adept at abstracting patterns from their training data, applying those patterns to existing language, and then producing language or a line of code that follows those patterns. Given the sheer scale of LLMs, they might generate a language or code sequence that doesn’t even exist yet. Just as you would review a colleague’s code, you should assess and validate AI-generated code, too.

  Why context matters for AI coding tools

  Developing good prompt crafting techniques is important because input code passes through something called a context window, which is present in all transformer-based LLMs. The context window represents the capacity of data an LLM can process. Though it can’t process an infinite amount of data, it can grow larger. Right now, the Codex model has a context window that allows it to process a couple of hundred lines of code, which has already advanced and accelerated coding tasks like code completion and code change summarization.

  Developers use details from pull requests, a folder in a project, open issues—and the list goes on—to contextualize their code. So, when it comes to a coding tool with a limited context window, the challenge is to figure out what data, in addition to code, will lead to the best suggestions.

  The order of the data also impacts a model’s contextual understanding. Recently, GitHub made updates to its pair programmer so that it considers not only the code immediately before the cursor, but also some of the code after the cursor. The paradigm—which is called Fill-In-the-Middle (FIM)—leaves a gap in the middle of the code for GitHub Copilot to fill, providing the tool with more context about the developer’s intended code and how it should align with the rest of the program. This helps produce higher quality code suggestions without any added latency.

  Visuals can also contextualize code. Multimodal LLMs (MMLLMs) scale transformer LLMs so they process images and videos, as well as text. OpenAI recently released its new GPT-4 model—and Microsoft revealed its own MMLLM called Kosmos-1. These models are designed to respond to natural language and images, like alternating text and images, image-caption pairs, and text data.

  GitHub’s senior developer advocate Christina Warren shares the latest on GPT-4 and the creative potential it holds for developers:

  Our R&D team at GitHub Next has been working to move AI past the editor with GitHub Copilot X. With this new vision for the future of AI-powered software development, we’re not only adopting OpenAI’s new GPT-4 model, but also introducing chat and voice, and bringing GitHub Copilot to pull requests, the command line, and docs. See how we’re investigating the future of AI-powered software development >

  How developers are using generative AI coding tools

  The field of generative AI is filled with experiments and explorations to uncover the technology’s full capabilities—and how they can enable effective developer workflows. Generative AI tools are already changing how developers write code and build software, from improving productivity to helping developers focus on bigger problems.

  While generative AI applications in software development are still being actively defined, today, developers are using generative AI coding tools to:

  • Get a head start on complex code translation tasks. A study presented at the 2021 International Conference on Intelligent User Interfaces found that generative AI provided developers with a skeletal framework to translate legacy source code into Python. Even if the suggestions weren’t always correct, developers found it easier to assess and fix those mistakes than manually translate the source code from scratch. They also noted that this process of reviewing and correcting was similar to what they already do when working with code produced by their colleagues.

  With GitHub Copilot Labs, developers can use the companion VS Code extension (that’s separate from but dependent on the GitHub Copilot extension) to translate code into different programming languages. Watch how GitHub Developer Advocate, Michelle Mannering, uses GitHub Copilot Labs to translate her Python code into Ruby in just a few steps.

  • Code more efficiently. While autocompletion has been in modern IDEs for years, LLMs can generate longer suggestions—sometimes multiple lines of code—that are often more relevant. A 2022 study published in the Proceedings of the Association for Computing Machinery on Programming Languages (PACMPL) observed 20 programmers who interacted with GitHub Copilot. They found that thanks to end-of-line suggestions for function calls and argument completions, developers were able to code faster and stay in the flow longer.

  Our own research supports these findings, too. As we mentioned earlier, we found that developers who used GitHub Copilot coded up to 55% faster than those who didn’t. But productivity gains went beyond speed with 74% of developers reporting that they felt less frustrated when coding and were able to focus on more satisfying work.

  • Tackle new problems and get creative. The PACMPL study also found that developers used GitHub Copilot to find creative solutions when they were unsure of how to move forward. These developers searched for next possible steps and relied on the generative AI coding tool to assist with unfamiliar syntax, look up the right API, or discover the correct algorithm.

  I was one of the developers who wrote GitHub Copilot, but prior to that work, I had never written a single line of TypeScript. That wasn’t a problem because I used the first prototype of GitHub Copilot to learn the language and, eventually, help ship the world’s first at-scale generative AI coding tool.

  - Albert Ziegler, Principal Machine Learning Engineer // GitHub

  • Find answers without leaving their IDEs. Some participants in the PACMPL study also treated GitHub Copilot’s multi-suggestion pane like StackOverflow. Since they were able to describe their goals in natural language, participants could directly prompt GitHub Copilot to generate ideas for implementing their goals, and press Ctrl/Cmd + Enter to see a list of 10 suggestions. Even though this kind of exploration didn’t lead to deep knowledge, it helped one developer to effectively use an unfamiliar API.

  A 2023 study published by GitHub in the Association for Computing Machinery’s Queue magazine also found that generative AI coding tools save developers the effort of searching for answers online. This provides them with more straightful forward answers, reduces context switching, and conserves mental energy.

  Part of GitHub’s new vision for the future of AI-powered software development is a ChatGPT-like experience directly in your editor. Watch how Martin Woodward, GitHub’s Vice President of Developer Relations, uses GitHub Copilot Chat to find and fix bugs in his code.

• 

  Generative AI-enabled compliance for software development
  • sorangutan  

  1
  1
  Posts
  9
  Views

  

  In our recent blog post announcing GitHub Copilot X, we mentioned that generative AI represents the future of software development. This amazing technology will enable developers to stay in the flow while helping enterprises meet their business goals.

  But as we have also mentioned in our blog series on compliance, generative AI may soon act as an enabler for developer-focused compliance programs that will drive optimization and keep your development, compliance and audit teams productive and happy.

  Today, we’ll explore the potential for generative AI to help enable teams to optimize and automate some of the foundational compliance components of separation of duties that many enterprises still often manage and review manually.

  Generative AI has been dominating the news lately—but what exactly is it? Here’s what you need to know, and what it means for developers.

  Separation of duties

  The concept of “separation of duties,” long used in the accounting world as a check and balance approach, is also adopted in other scenarios, including technology architecture and workflows. While helpful to address compliance, it can lead to additional manual steps that can slow down delivery and innovation.

  Fortunately, the PCI-DSS requirements guide provides a more DevOps, cloud native, and AI-enabled approach to separation of duties by focusing on functions and accounts, as opposed to people:

  “The purpose of this requirement is to separate the development and test functions from the production functions. For example, a developer can use an administrator-level account with elevated privileges in the development environment and have a separate account with user-level access to the production environment.”

  There are many parts of a software delivery workflow that need to have separation of duties in place—but one of the core components that is key for any compliance program is the code review. Having a separate set of objective eyes reviewing your code, whether it’s human or AI-powered, helps to ensure risks, tech debt, and security vulnerabilities are found and mitigated as early as possible.

  Code reviews also help enable the concept of separation of duties since it prohibits a single person or a single function, account, or process from moving code to the next part of your delivery workflow. Additionally, code reviews help enable separation of duties for Infrastructure as Code (IaC) workflows, Policy-as-Code configurations, and even Kubernetes declarative deployments.

  As we mentioned in our previous blog, GitHub makes code review easy, since pull requests are part of the existing workflow that millions of developers use daily. Having a foundational piece of compliance built-in to the platform that developers know and love keeps them in the flow, while keeping compliance and audit teams happy as well.

  Generative AI and pull requests

  Wouldn’t it be cool if one-day generative AI could be leveraged to enable more developer-friendly compliance programs which have traditionally been very labor and time intensive? Imagine if generative AI could help enable DevOps and cloud native approaches for separation of duties by automating tedious tasks and allowing humans to focus on key value-added tasks.

  Bringing this back to compliance and separation of duties, wouldn’t it be great if a generative AI helper was available to provide an objective set of eyes on your pull requests? This is what the GitHub Next team has been working towards with GitHub Copilot for Pull Requests.

  • Suggestions for your pull request descriptions. AI-powered tags are embedded into a pull request description and automatically filled out by GitHub Copilot based on the code which the developers changed. Going one step further, the GitHub Next team is also looking at the creation of descriptive sentences and paragraphs as developers create pull requests.

  • Code reviews with AI. Taking pull requests and code reviews one step further, the GitHub Next team is looking at AI to help review the code and provide suggestions for changes. This will help enable human interactions and optimize existing processes. The AI would automate the creation of the descriptions, based on the code changes, as well as suggestions for improvements. The code reviewer will have everything they need to quickly review the change and decide to either move forward or send the change back.

  When these capabilities are production ready, development teams and compliance programs will appreciate these features for a few reasons. First, the pull request and code review process would be driven by a conversation based on a neutral and independent description. Second, the description will be based on the actual code that was changed. Third, both development and compliance workflows will be optimized and allow humans to focus on value-added work.

  While these capabilities are still a work in progress, there are features available now that may help enable compliance, audit, and security teams with GitHub Copilot for Business. The ability for developers to complete tasks faster and stay in the flow are truly amazing. But the ability for GitHub Copilot to provide AI-based security vulnerability filtering nowis a great place for compliance and audit teams within enterprises to get started on their journey to embracing generative AI into their day-to-day practices.

  Next steps

  Generative AI will enable developers and enterprises to achieve success by reducing manual tasks and enabling developers to focus their creativity on business value–all while staying in the flow.

  I hope this blog will help drive positive discussions regarding this topic and has provided a forward looking view into what will be possible in the future. The future ability of generative AI to help enable teams by automating tedious tasks will help humans focus on more value-added work and could eventually be an important part of a robust and risk-based compliance posture.

  Get GitHub Copilot

   

  Explore GitHub Copilot X >

  
  
  https://github.blog/2023-04-11-generative-ai-enabled-compliance-for-software-development/
• 

  How enabling developers can help drive financial inclusion
  • sorangutan  

  1
  1
  Posts
  23
  Views

  

  Developers who feel more satisfied in their jobs are better positioned to be more productive. We also know developers can gain a sense of fulfillment by making an impact beyond the walls of their company and elevating their community. An opportunity exists, which developers can meet, to support those who lack access to the financial system. Many countries are working to drive financial inclusion through different motions—to which developers can contribute. GitHub provides a set of tools and services, which can support your developers working to address this need.

  For example, in Australia, there is a huge opportunity to continue the work aimed at reaching those who are not currently included in the financial system. There are still a large number of people that don’t have access to important services that many of us take for granted—an opportunity that financial inclusion tries to solve.

  Let’s explore these opportunities and how GitHub can help.

  Financial inclusion explained

  The World Bank defines financial inclusion as providing individuals and businesses access to affordable financial products to meet their needs. This includes products, such as checking accounts, credit cards, mortgages, and payments, which are still not available to over a billion unbanked people around the world. Many of these are women, people living in poverty, and those living outside of large cities.

  Open Finance (or Open Banking) is an approach adopted by banks like NAB (National Australia Bank)to help include more individuals in the financial system by providing them access to the best products and services in a secure way that addresses their needs.

  To enable financial inclusion and Open Finance, there needs to be a channel to exchange data and services between banks, customers, and trusted partners (fintechs, for example); that is where application programming interfaces (APIs) come in. The easiest way to understand an API is to think of it as a contract between two applications that need a standardized and secure way to talk to each other. Once the contract is created and secured, it can be used anywhere to share data or initiate a financial transaction.

  This API-driven innovation lowers barriers for those individuals who may have limited physical access to banks, credit cards, or traditional financial products.

  How GitHub can help

  The tremendous opportunities for Australia, New Zealand, India, and other countries to enable financial inclusion to its population are dependent on the quality of the APIs. The quality and adoption of the APIs is dependent on creating a great developer experience because they are the ones building the APIs and applications that will leverage them.

  GitHub is used by 100 million developers and is widely-recognized as the global home of open source. Developer experience is at the core of everything we do and it empowers developers to do their best work and be happy. But how does GitHub help enable financial inclusion and Open Finance?

  The Open Bank Project released a report in 2020 highlighting how providing a great developer experience can drive growth of APIs that enable financial inclusion. Several topics which were highlighted and where GitHub can help are:

  1. Create solutions to help people

  This is an important motivator for developers. If developers create solutions that can help increase financial inclusion, they should make sure those solutions are available to as many people as possible through the open source community. Since we know that open source is the foundation of more than 90% of the world’s software, there is a great opportunity to collaborate globally and build on solutions that already exist.

  Because GitHub is the home of open source and has 100 million developers, there is no better place for developers to create solutions that will make the biggest impact.

  2. Running Hackathons

  Hackathons, like the Global Open Finance Challenge, which NAB collaborated in and was won by an Aussie start-up, are important for developers to share ideas with other developers and large enterprises. They help developers see what APIs are currently available and enable innovation and collaboration with a global reach. To run a successful hackathon, developers will need to have access to code and documentation, which has been open sourced—and GitHub is a key component to enable this.

  3. Recognition for developers

  If a developer has worked on a solution that is helping enable financial inclusion, it’s important to ensure their effort is recognized and supported. The most important part of recognizing the awesome work developers do is to make sure there is a single platform where this work can be shared. Thankfully, that platform already exists and any developer, anywhere in the world, can access it for free—it’s GitHub!

  Tip: Is there a project on GitHub that you rely on? Consider giving the repository a star, or creating a a new GitHub Discussion to let the maintainer know you’re thankful!

  At GitHub, we also know that sometimes recognition isn’t enough, and developers need support. This is why the GitHub Sponsors program was created. We also created our GitHub for Startups program which provides support to the startup community around the world—many of whom are important contributors to Open Banking.

  4. Documentation

  The success of an API is dependent on how easy it is for developers to understand and use. If developers are unable to quickly understand the context of the API, how to connect to it, or easily set it up to test it, then it probably won’t be successful.

  The topic of API documentation and API Management is beyond the scope of this post, but it’s important to remember that open source is a key enabler of Open Finance and developers will need a platform to collaborate and share documentation and code. GitHub is the best platform for that, and we have seen at least a 50% increase in productivity when developers leverage documentation best practices enabled by GitHub.

  Call to action

  Developers have an amazing opportunity to contribute to the financial inclusion work that is happening in Australia and across the world. GitHub can help support developers to address this opportunity by giving them the tools and services they need to be productive and successful.

  We’ve recently launched our weekly livestream on LinkedIn Live, GitHub in my Day Job, for those who want to learn more about how GitHub empowers developers across the community while providing guardrails to govern, and remain compliant. So, join us at https://gh.io/dayjob—we can’t wait to have you with us.

  
  
  https://github.blog/2023-04-10-how-enabling-developers-can-help-drive-financial-inclusion/
• 

  What developers need to know about generative AI
  • sorangutan  

  1
  1
  Posts
  17
  Views

  

  By now, you’ve heard of generative artificial intelligence (AI) tools like ChatGPT, DALL-E, and GitHub Copilot, among others. They’re gaining widespread interest thanks to the fact that they allow anyone to create content from email subject lines to code functions to artwork in a matter of moments.

  This potential to revolutionize content creation across various industries makes it important to understand what generative AI is, how it’s being used, and who it’s being used by. In this article, we’ll explore what generative AI is, how it works, some real-world applications, and how it’s already changing the way people (and developers) work.

  What is generative AI used for?

  You may have heard the buzz around new generative AI tools like ChatGPT or the new Bing, but there’s a lot more to generative AI than any one single framework, project, or application.

  Traditional AI systems are trained on large amounts of data to identify patterns, and they’re capable of performing specific tasks that can help people and organizations. But generative AI goes one step further by using complex systems and models to generate new, or novel, outputs in the form of an image, text, or audio based on natural language prompts.

  Generative AI models and applications can, for example, be used for:

  • Text generation. Text generation, as a field, with AI tools has been in development since the 1970s—but more recently, AI researchers have been able to train generative adversarial networks (GANs) to produce text that models human-like speech. A prime example is OpenAI’s application ChatGPT, which has been trained on thousands of texts, books, articles, and code repositories, and can respond with full answers to natural language prompts and questions.

  An example of text generation in ChatGPT

  • Image generation. Generative AI models can be used to create new images with natural language prompts, which is one of the most popular techniques with current tools and applications. The goal with text-to-image generation is to create an image that accurately represents the content of a given prompt. For example, when we give the text prompt, “impressionist style oil painting of a Shiba Inu dog giving a tarot card reading,” to the popular AI image generator DALL-E 2 we get something that looks like this (and yes, it’s a gem):

  An AI-generated image from DALL-E 2 of a Shiba Inu dog giving a tarot card reading

  • Video generation. Generative AI models, like Stable Diffusion, are creating new videos from existing videos by applying specified styles through a text prompt or image reference. One project on GitHub, stable-diffusion-videos, offers helpful examples and tips on how to create music videos and videos that can morph between text prompts with Stable Diffusion.

  

  • Programming code generation. Rather than scouring the internet or developer community groups for help with code examples, generative AI models can be used to help generate new programming code with natural language prompts, complete partially written code with suggestions, or even translate code from one programming language to another. This is how, at a simple level, GitHub Copilot works: it uses OpenAI’sCodex model to offer code suggestions right from a developer’s editor. However, as you would with any software development tool, we encourage you to review generated code before merging into production.

  • Data generation. Creating new data—which is called synthetic data—and augmenting existing data sets is another common use case for generative AI. This involves generating new samples from an existing dataset to increase the dataset’s size and improve machine learning models trained on it, all while providing a layer of privacy since real user data is not being utilized to power models. Synthetic data generation provides a way to create useful, meaningful data for more than just ML training though—a number of self-driving car companies like Cruise and Waymo utilize AI-generated synthetic data for training perception systems to prepare vehicles for real-world situations while in operation.

  • Language translation. Natural-language understanding (NLU) models combined with generative AI have become increasingly popular to provide language translations on-the-fly. These types of tools help companies break language barriers and increase their scope of accessibility for customer bases by being able to provide things like support or documentation in their native language. Through complex, deep learning algorithms, generative AI is able to understand the context of a source text and linguistically construct those sentences in another language. This practice can also apply to coding languages, for example, translating a desired function from Python to Java.

  The bottom line: Even though generative AI is a relatively new technology, it’s already being used in consumer and business applications. The use cases, as well as the quantity of applications created with it, will continue evolving to meet more distinct and specific needs.

  How does generative AI work?

  Generative AI models work by using neural networks to identify patterns from large sets of data, then generate new and original data or content.

  But what are neural networks? In simple terms, they use interconnected nodes that are inspired by neurons in the human brain. These networks are the foundation of machine learning and deep learning models, which use a complex structure of algorithms to process large amounts of data such as text, code, or images. Training these neural networks involves adjusting the weights or parameters of the connections between neurons to minimize the difference between predicted and desired outputs, which allows the network to learn from mistakes and make more accurate predictions based on the data.

  Algorithms are a key component of machine learning and generative AI models. But beyond helping machines learn from data, algorithms are also used to optimize accuracy of outputs and make decisions, or recommendations, based on input data.

  While algorithms help automate these processes, building a generative AI model is incredibly complex due to the massive amounts of data and compute resources they require. People and organizations need large datasets to train these models, and generating high-quality data can be time-consuming and expensive.

  To restate the obvious, these models are complicated. Need proof? Here are some common generative AI models and how they work:

  • Large language models (LLM): LLMs are a type of machine learning model that process and generate natural language text. One of the most significant advancements in the development of large language models has been the availability of vast amounts of text data, such as books, websites, and social media posts. This data can be used to train models that are capable of predicting and generating natural language responses in a variety of contexts. As a result, large language models have multiple practical applications, such as virtual assistants, chatbots, or text generators, like ChatGPT.

  • Generative adversarial networks (GAN): GANs are one of the most used models for generative AI, and they employ two different neural networks. GANs consist of two different types of neural networks: a generator and a discriminator. The generator network generates new data, such as images or audio, from a random noise signal while the discriminator is trained to distinguish between real data from the training set and the data produced by the generator.

  During training, the generator tries to create data that can trick the discriminator network into thinking it’s real. This “adversarial” process will continue until the generator can produce data that is totally indistinguishable from real data in the training set. This process helps both networks improve at their respective tasks, which ultimately results in more realistic and higher-quality generated data.

  A diagram illustrating how a generative adversarial network works. Image [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/deed.en) האדם-החושב on wikipedia

  • Transformer-based models: A transformer-based model’s neural networks operate by learning context and meaning through tracking relationships of sequential data, which means these models are really good at natural language processing tasks like machine translation, language modeling, and answering questions. These models have been used in popular language models, such as GPT-4 (which stands for Generative Pre-trained Transformer 4), and have also been adapted for other such tasks that require the modeling of sequential data such as image recognition.

  • Variational autoencoder models (VAEs): These models are similar to GANs in that they work with two different neural networks: encoders and decoders. VAEs can take a large amount of data and compress it into a smaller representation, which can be used to create new data that is similar to the original data. VAEs are often used in image, video, and audio generation—and here’s a fun fact: you can train a VAE on datasets like CelebA, which contains over 200,000 images of celebrities, to create completely new portraits of people that don’t exist.

  The smile vector, a concept vector discovered by Tom White using VAEs trained on the CelebA dataset.

  The real-world applications of generative AI

  The impact of generative AI is quickly becoming apparent—but it’s still in its early days. Despite this, we’re already seeing a proliferation of applications, products, and open source projects that are using generative AI models to achieve specific outcomes for people and organizations (and yes, developers, too).

  Though generative AI is constantly evolving, it already has some solid real world applications. Here’s just a few of them:

  Coding

  New and seasoned developers alike can utilize generative AI to improve their coding processes. Generative AI coding tools can help automate some of the more repetitive tasks, like testing, as well as complete code or even generate brand new code. GitHub has its own AI-powered pair programmer, GitHub Copilot, which uses generative AI to provide developers with code suggestions. And GitHub also has announced GitHub Copilot X, which brings generative AI to more of the developer experience across the editor, pull requests, documentation, CLI, and more.

  Accessibility

  Generative AI has the potential to greatly impact and improve accessibility for folks with disabilities through a variety of modalities, such as speech-to-text transcription, text-to-speech audio generation, or assistive technologies. One of the most exciting facets of our GitHub Copilot tool is its voice-activated capabilities that allow developers with difficulties using a keyboard to code with their voice. By leveraging the power of generative AI, these types of tools are paving the way for a more inclusive and accessible future in technology.

  Gaming

  Generative AI can take gaming to the next level (get it? ) by generating new characters, storylines, design components, and more. Case in point: The developer behind the game, This Girl Does Not Exist, has said that every component of the game—from the storyline to the art and even the music—was generated entirely by AI. This use of generative AI can enable gaming studios to create new and exciting content for their users, all without increasing the developer workload, which frees them up to work on other aspects of the game, such as story development.

  Web design

  Designers can utilize generative AI tools to automate the design process and save significant time and resources, which allows for a more streamlined and efficient workflow. Additionally, incorporating these tools into the development process can lead to the creation of highly customized designs and logos, enhancing the overall user experience and engagement with the website or application. Generative AI tools can also be used to do some of the more tedious work, such as creating design layouts that are optimized and adaptable across devices. For example, designers can use tools like designs.ai to quickly generate logos, banners, or mockups for their websites.

  Web search

  Microsoft and other industry players are increasingly utilizing generative AI models in search to create more personalized experiences. This includes query expansion, which generates relevant keywords to reduce the number of searches. So, rather than the search engine returning a list of links, generative AI can help these new and improved models return search results in the form of natural language responses. Bing now includes AI-powered features in partnership with OpenAI that provide answers to complex questions and allow users to ask follow-up questions in a chatbox for more refined responses.

  Healthcare

  Interest has emerged around the potential applications of generative AI in the healthcare field to improve disease detection and diagnosis, advance medical research, and accelerate progress in the pharmaceutical space. Potentially, generative AI could be used to analyze large amounts of data to simulate chemical structures and predict new compounds will be the most effective for new drug discoveries. NVIDIA Clara is one example of a generative AI model specifically designed for medical imaging and healthcare research. (Plus, Gartner suggests more than 30 percent of new pharmaceutical drugs and materials will be discovered via generative AI models by 2025.)

  Fun fact: Did you know that ChatGPT recently passed the US Medical Licensing exam without any intervention from clinicians?

  Marketing and advertising

  In marketing, content is king—and generative AI is making it easier than ever to quickly create large amounts of it. A number of companies, agencies, and creators are already turning to generative AI tools to create images for social posts or write captions, product descriptions, blog posts, email subject lines, and more. Generative AI can also help companies personalize ad experiences by creating custom, engaging content for individuals at speed. Writers, marketers, and creators can leverage tools like Jasper to generate copy, Surfer SEO to optimize organic search, or albert.ai to personalize digital advertising content.

  Art and design

  As we’ve seen above, the power of AI can be harnessed to create some incredible portraits in a matter of moments (re: the future-telling Shiba ). Artists and designers alike are using these AI tools as a source of inspiration. For example, architects can quickly create 3D models of objects or environments and artists can breathe new life into their portraits by using AI to apply different styles, like adding a Cubist style to their original image. Need proof? Designers are already starting to use AI image generators, such as Midjourney and Microsoft Designer, to create high-quality images by simply typing out Discord commands.

  Finance

  In a recent discussion about tech trends and how they’ll affect the finance sector, Michael Schrage, a research fellow at the MIT Sloan School Initiative on the Digital Economy, said, “I think, increasingly, we’re going to be seeing generative AI used for financial forecasts and scenario generation.” This is a likely path forward—generative AI can be used to analyze large amounts of data to detect fraud, manage risk, and inform decision making. And that has obvious applications in the financial services industry.

  Manufacturing

  Manufacturers are starting to turn to generative AI solutions to help with product design, quality control, and predictive maintenance. Generative AI can be used to analyze historical data to improve machine failure predictions and help manufacturers with maintenance planning. According to research conducted by Capgemini, more than half of European manufacturers are implementing some AI solutions (although so far, these aren’t generative AI solutions). This is largely because the sheer amount of manufacturing data is easier for machines to analyze at speed than humans.

  AI as a partner: Generative AI models and tools are narrow in focus, and work best at generating content, code, and images. In research at GitHub, we’ve found that GitHub Copilot helps developers code up to 55% faster, underscoring how generative AI models and tools can improve overall productivity and boost efficiency. Metrics like these show how generative AI tools are already changing how people and teams work—but they also underscore how these tools act as complement to human efforts.

  Take this with you

  Whether it’s creating visual assets for an ad campaign or augmenting medical images to help diagnose diseases, generative AI is helping us solve complex problems at speed. And the emergence of generative AI-based programming tools has revolutionized the way developers approach writing code.

  We know that developers want to design and write software quickly, and tools like GitHub Copilot are enabling them to access large datasets to write more efficient code and boost productivity. In fact, 96% of developers surveyed reported spending less time on repetitive tasks using GitHub Copilot, which in turn allowed 74% of them to focus on more rewarding work.

  While these models aren’t perfect yet, they’re getting better by the day—and that’s creating an exciting immediate future for developers and generative AI.

  
  
  https://github.blog/2023-04-07-what-developers-need-to-know-about-generative-ai/
• 

  Building GitHub with Ruby and Rails
  • sorangutan  

  1
  1
  Posts
  9
  Views

  

  Since the beginning, GitHub.com has been a Ruby on Rails monolith. Today, the application is nearly two million lines of code and more than 1,000 engineers collaborate on it daily. We deploy as often as 20 times a day, and nearly every week one of those deploys is a Rails upgrade.

  Upgrading Rails weekly

  Every Monday a scheduled GitHub Action workflow triggers an automated pull request, which bumps our Rails version to the latest commit on the Rails main branch for that day. All our builds run on this new version of Rails. Once all the builds pass, we review the changes and ship it the next day. Starting an upgrade on Monday you will already have an open pull request linking the changes this Rails upgrade proposes and a completed build.

  This process is a far stretch from how we did Rails upgrades only a few years ago. In the past, we spent months migrating from our custom fork of Rails to a newer stable release, and then we maintained two Gemfiles to ensure we’d remain compatible with the upcoming release. Now, upgrades take under a week. You can read more about this process in this 2018 blog post. We work closely with the community to ensure that each Rails release is running in production before the release is officially cut.

  There are real tangible benefits to running the latest version of Rails:

  • We give developers at GitHub the very best version of our tools by providing the latest version of Rails. This ensures users can take advantage of all the latest improvements including better database connection handling, faster view rendering, and all the amazing work happening in Rails every day.
  • We have removed nearly all of our Rails patches. Since we are running on the latest version of Rails, instead of patching Rails and waiting for a change, developers can suggest the patch to Rails itself.
  • Working on Rails is now easier than ever to share with your team! Instead of telling your team you found something in Rails that will be fixed in the next release, you can work on something in Rails and see it the following week!
  • Maintaining more up-to-date dependencies gives us a better security posture. Since we already do weekly upgrades, adding an upgrade when there is a security advisory is standard practice and doesn’t require any extra work.
  • There are no “big bang” migrations. Since each Rails upgrade incorporates only a small number of changes, it’s easier to understand and dig into if there are incompatibilities. The worst issues from a tough upgrade are unexpected changes from an unknown location. These issues can be mitigated by this upgrade strategy.
  • Catching bugs in the main branch and contributing back strengthens our engineering team and helps our developers deepen their expertise and understanding of our application and its dependencies.

  Testing Ruby continuously

  Naturally, we have a similar process for Ruby upgrades. In February 2022, shortly after upgrading to Ruby 3.1, we started building and testing Ruby shas from 3.2-alpha in a parallel build. When CI runs for the GitHub Rails application, two versions of the builds run: one build uses the Ruby version we are running in production and one uses the latest Ruby commit including the latest changes in Ruby, which we update weekly.

  While we build Ruby with every change, GitHub only ships numbered Ruby versions to production. The builds help us maintain compatibility with the upcoming Ruby version and give us insight into what Ruby changes are coming.

  In early December 2022, with CI giving us confidence we were compatible before the usual Christmas release of Ruby 3.2, we were able to test Ruby release candidates with a portion of production traffic and give the Ruby team insights into any changes we noticed. For example, we could reproduce an increase in allocations due to keyword argument handling that was fixed before the release of Ruby 3.2 due to this process. We also identified a subtle change when to_str and #to_i is applied. Because we upgrade all the time, identifying and resolving these issues was standard practice.

  This weekly upgrade process for Ruby allowed us to upgrade our monolith from Ruby 3.1 to Ruby 3.2 within a month of release. After all, we had already tested and run it in production! At this point, this was the fastest Ruby upgrade we had ever done. We broke this record with the release of Ruby 3.2.1, which we adopted on release day.

  This upgrade process has proved to be invaluable for our collaboration with the Ruby core team. A nice side effect of having these builds is that we are able to easily test and profile our own Ruby changes before we suggest them upstream. This can make it easier for us to identify regressions in our own application and better understand the impact of changes on a production environment.

  Should I do it, too?

  Our ability to do frequent Ruby and Rails upgrades is due to some engineering maturity at GitHub. Doing weekly Rails upgrades requires a thorough test suite with many great engineers working to maintain and improve it. We also gain confidence from having great test environments along with progressive rollout deploys. Our test suite is likely to catch problems, and if it doesn’t, we are confident we will catch it during deploy before it reaches customers.

  If you have these tools, you should also upgrade Rails weekly and test using the latest Ruby. GitHub is a better Rails app because of it and it has enabled work from my team that I am really proud of.

  Ruby champion Eileen Uchitelle explains why investing in Rails is important in her Rails Conf 2022 Keynote:

  Ultimately, if more companies treated the framework as an extension of the application, it would result in higher resilience and stability. Investment in Rails ensures your foundation will not crumble under the weight of your application. Treating it as an unimportant part of your application is a mistake and many, many leaders make this mistake.

  Thanks to contributions from people around the world, using Ruby is better than ever. GitHub, along with hundreds of other companies, benefits from Ruby and Rails continuing to improve. Upgrading regularly and investing in our frameworks is a staple of the work we do on the Ruby Architecture team at GitHub. We are always grateful for the Ruby community and glad that we can give back in a way that improves our application and tools as much as it improves them for everyone else.

  
  
  https://github.blog/2023-04-06-building-github-with-ruby-and-rails/
• 

  GitHub Availability Report: March 2023
  • sorangutan  

  1
  1
  Posts
  15
  Views

  

  In March, we experienced six incidents that resulted in degraded performance across GitHub services. This report also sheds light into a February incident that resulted in degraded performance for GitHub Codespaces.

  February 28 15:42 UTC (lasting 1 hour and 26 minutes)

  On February 28 at 15:42 UTC, our monitors detected a higher than normal failure rate for creating and starting GitHub Codespaces in the East US region, caused by slower than normal VM allocation time from our cloud provider. To reduce impact to our customers during this time, we redirected codespace creations to a secondary region. Codespaces in other regions were not impacted during this incident.

  To help reduce the impact of similar incidents in the future, we tuned our monitors to allow for quicker detection. We are also making architectural changes to enable failover for existing codespaces and to initiate failover automatically without human intervention.

  March 01 12:01 UTC (lasting 3 hour and 06 minutes)

  On March 1 at 12:01 UTC, our monitors detected a higher than normal latency for requests to Container, NPM, Nuget, and RubyGems Packages registries. Initial impact was 5xx responses to 0.5% of requests, peaking at 10% during this incident. We determined the root cause to be an unhealthy disk on a VM node hosting our database, which caused significant performance degradation at the OS level. All MySQL servers on this problematic node experienced connection delays and slow query execution, leading to the high failure rate.

  We mitigated this with a failover of the database. We recognize this incident lasted too long and have updated our runbooks for quicker mitigation short term. To address the reliability issue, we have architectural changes in progress to migrate the application backend to a new MySQL infrastructure. This includes improved observability and auto-recovery tooling, thereby increasing application availability.

  March 02 23:37 UTC (lasting 2 hour and 18 minutes)

  On March 2 at 23:37 UTC, our monitors detected an elevated amount of failed requests for GitHub Actions workflows, which appeared to be caused by TLS verification failures due to an unexpected SSL certificate bound to our CDN IP address. We immediately engaged our CDN provider to help investigate. The root cause was determined to be a configuration change on the provider’s side that unintentionally changed the SSL certificate binding to some of our GitHub Actions production IP addresses. We remediated the issue by removing the certificate binding.

  Our team is now evaluating onboarding multiple DNS/CDN providers to ensure we can maintain consistent networking even when issues arise that are out of our control.

  March 15 14:07 UTC (lasting 1 hour and 20 minutes)

  On March 15 at 14:07 UTC, our monitors detected increased latency for requests to Container, NPM, Nuget, RubyGems package registries. This was caused by an operation on the primary database host that was performed as part of routine maintenance. During the maintenance, a slow running query that was not properly drained blocked all database resources. We remediated this issue by killing the slow running query and restarting the database.

  To prevent future incidents, we have paused maintenance until we investigate and address the draining of long running queries. We have also updated our maintenance process to perform additional safety checks on long running queries and blocking processes before proceeding with production changes.

  March 27 12:25 UTC (lasting 1 hour and 33 minutes)

  On March 27 at 12:25 UTC, we were notified of impact to pages, codespaces and issues. We resolved the incident at 13:29 UTC.

  Due to the recency of this incident, we are still investigating the contributing factors and will provide a more detailed update in next month’s report.

  March 29 14:21 UTC (lasting 4 hour and 29 minutes)

  On March 29 at 14:21 UTC, we were notified of impact to pages, codespaces and actions. We resolved the incident at 18:50 UTC.

  Due to the recency of this incident, we are still investigating the contributing factors and will provide a more detailed update in next month’s report.

  March 31 01:16 UTC (lasting 52 minutes)

  On March 31 at 01:16 UTC, we were notified of impact to Git operations, issues and pull requests. We resolved the incident at 02:08 UTC.

  Due to the recency of this incident, we are still investigating the contributing factors and will provide a more detailed update in next month’s report.

  ---

  Please follow our status page for real-time updates on status changes. To learn more about what we’re working on, check out the GitHub Engineering Blog.

  
  
  https://github.blog/2023-04-05-github-availability-report-march-2023/
• 

  How the Grafana Alerting team scales their issue management with GitHub Projects
  • sorangutan  

  1
  1
  Posts
  24
  Views

  

  At GitHub you’ve heard us talk about how we are using GitHub Projects and GitHub Actions to plan and track our work and now we’ve asked one of our customers, Grafana Labs, to share how their teams are approaching work in a new way. Whether they are managing open source requests, operational tasks, or escalations, the Grafana Labs Alerting team uses GitHub Projects to manage all these issues efficiently.

  Let’s hear more from Armand Grillet, Senior Engineering Manager at Grafana Labs, including how his teams use tasklists to break work into manageable tasks, use a common set of labels to filter tasks, create multiple views on a single project to meet the needs of different teams and stakeholders, and use automation to enable engineers to stay focused on the code.

  ---

  Grafana is a leading open source platform for monitoring and observability, which is why the Grafana Labs GitHub organization is the center of our engineering efforts with nearly 1,000 repositories, including six having more than 2,000 stars. In addition to this open source work, Grafana Labs engineers take care of the Grafana Cloud observability platform and its customers’ escalations.

  As the manager of the Grafana Alerting and service-level objectives (SLOs) backend teams at Grafana Labs it was essential to have one project board that benefited our multiple stakeholders: team members, other employees, and the open source community. Our GitHub Project has offered us the opportunity to do just that. You can even make a copy of our board and adapt it for your own project needs, using the ‘make a copy’ functionality.

  One view for each team, assembled around task lists

  Our four teams each have a view in the Grafana Alerting project—”Backend,” “Front-end,” “UX,” and “Docs.”

  Grafana Alerting has contributors working across four teams: backend, front-end, UX, and docs. Each of these contributor types has their own view in our project. The field options (thus the columns) are the same for each of these views:

  • Inbox—not reviewed yet
  • Waiting for input—open source issues that need more details
  • Backlog—reviewed, priority depending on the milestone set
  • In progress
  • In review
  • Done

  These columns (custom fields) are intentionally generic. They can work for all teams, no matter whether the issue has been written by someone in the community or by someone internally. We use project filters based on labels, such as “area/frontend” to allow issues to be automatically added to the correct views once they are added to our project.

  Big issues that we work on over a quarter use tasklists to breakdown the main pieces of work.

  Note: Tasklists are currently in private beta; you can sign your organization up for access on the GitHub Feature Preview portal

  For bigger issues, we make use of the GitHub tasklist feature to break down the work into tasks. We use labels to filter the tasks to be included in the relevant team’s view. This creates views that provide two different, but useful, kinds of information. For example, for the docs team:

  • Smaller issues with only the label “type/docs” are items the docs team needs to work on.
  • Bigger issues containing task lists with the label “type/docs” along with labels for additional areas (for example, “area/frontend”) are issues with a dependency on the Docs team, but the item is not owned by “docs.” For these bigger issues, if the status is “in progress” someone in the docs team should start checking how they can help.

  As a manager of the backend team and project lead for Grafana Alerting, this workflow gives me peace of mind. Even if our docs team members miss some meetings, the docs view in our project will always be accurate, since the engineers maintain the status of issues.

  The four team views, combined with our “Epics” view (the first view in our board) that lists our big issues for an entire quarter, allow everyone to see our progress on Grafana Alerting. GitHub users who are part of the Grafana Labs organization can see all issues, whereas Github users within our community can only see issues in public repositories. As most of our issues live in the Grafana public repositorythis allows us to be transparent by default.

  Whilst we use a private repository for issues relating to our operational work, we use the same labels in all our alerting-related repositories so that we can use project filters easily. Having common labels in many repositories creates an incentive to have the same labels in other repositories, especially new ones, even if they do not relate to alerting. This growing commonality makes searching for issues across multiple repositories easier, which is particularly useful for our product managers.

  Custom fields to create tailored views

  A valuable feature of GitHub Projects is the ability to have different columns (custom fields) per view. This allows us to view not only smaller issues but also larger issues covering an entire quarter. Our team also handles engineering escalations that are worked at a faster pace compared to normal issues.

  As a team, our three custom fields are:

  • Status: the default field for all issues except escalations (as mentioned above).
  • Quarter: used for bigger issues that include tasklists.
  • Escalation: used to capture more granular status of each escalation.

  With these custom fields, we can have custom views, such as our epic view, which gives us a birds-eye look of our quarterly goals, and our escalations view, which lets us review the state of escalations that need engineering work.

  The escalations view uses the escalation status custom field with special values such as “Waiting for release.”

  Thanks to the adaptability of GitHub Projects, we finally have one ‘source-of-truth’ to reference normal issues, big issues, and escalations.

  Enhancing projects with GitHub Actions

  For escalations, which are urgent to solve, we also use GitHub Actions to notify us on Slack or use Grafana OnCall if this is a high priority escalation.

  Combining GitHub Projects and GitHub Actions offers endless possibilities. Actions like github/issue-labeler allow us to have repositories shared by different teams with issues labeled automatically depending on keywords used in an issue. These labels are then used by other actions to add the issue to the right view or to send notifications to external systems. Opening the project and seeing that new relevant issues have been added automatically, ready for triage, feels like magic.

  An issue labeled “alerting” and “prio/3” has been created by support in “grafana/support-escalations.” This adds the issue to the right project and notifies the current on-call on Slack via GitHub Actions.

  We often have escalations related to features requested by customers. Being able to link to an escalation in an open source issue allows engineers and product managers to prioritize together on one platform while keeping this information confidential from GitHub users outside our organization.

  These automations have been important in terms of engineers’ motivation and our productivity. For example, the Grafana Alerting team receives around five issues from support regarding escalations per week. During the last quarter of 2022, at any given time, the team had on average only four open escalations due to the automation motivating engineers to prioritize this work. This is a significant reduction from the average of 20 open escalations at any given time prior to our shift to GitHub Projects and GitHub Actions.

  Combining GitHub Projects and GitHub Actions allows us to route feedback from various sources through the most relevant repos and then onto a single project.

  GitHub Projects at scale

  Since we started using GitHub Projects, our team has been growing. An important moment for me in our scaling was the creation of a new project for the team working on SLOs. Originally, this team had no escalations and a small number of engineers. We were able to easily make a copy of the ‘Alerting’ project, changing the filters to accommodate the new team’s repositories, too. The important question was whether the workflows we had for a big team would be too much for such a new team. The answer was yes, there were too many views, creating unnecessary overhead, but it was very easy to reduce the number of views to adapt to the new team’s needs. Using the existing project as a starting point for the SLO team enabled us to get a new project up and running fast.

  After a year of using GitHub Projects, we have seen its ability to handle our different types of issues and to adapt to the needs of our various teams at Grafana Labs. We find GitHub Projects to be a flexible planning and tracking solution that enables engineers to stay focussed on code and gives issues more visibility. GitHub Projects keeps the burden for contributors low whilst still providing the views managers need to understand and plan team efforts. GitHub Projects is the only tool the Grafana Alerting team needs to do project planning.

  
  
  https://github.blog/2023-03-15-how-the-grafana-alerting-team-scales-their-issue-management-with-github-projects/
• 

  Highlights from Git 2.40
  • sorangutan  

  1
  1
  Posts
  20
  Views

  

  The open source Git project just released Git 2.40 with features and bug fixes from over 88 contributors, 30 of them new.

  We last caught up with you on the latest in Git when 2.39 was released. To celebrate this most recent release, here’s GitHub’s look at some of the most interesting features and changes introduced since last time.

  ---

  • Longtime readers will recall our coverage of git jump from way back in our Highlights from Git 2.19 post. If you’re new around here, don’t worry: here’s a brief refresher.

    git jump is an optional tool that ships with Git in its contrib directory. git jump wraps other Git commands, like git grep and feeds their results into Vim’s quickfix list. This makes it possible to write something like git jump grep foo and have Vim be able to quickly navigate between all matches of “foo” in your project.

    git jump also works with diff and merge. When invoked in diff mode, the quickfix list is populated with the beginning of each changed hunk in your repository, allowing you to quickly scan your changes in your editor before committing them. git jump merge, on the other hand, opens Vim to the list of merge conflicts.

    In Git 2.40, git jump now supports Emacs in addition to Vim, allowing you to use git jump to populate a list of locations to your Emacs client. If you’re an Emacs user, you can try out git jump by running:

    M-x grepgit jump --stdout grep foo

    [source]

  • If you’ve ever scripted around a Git repository, you may be familiar with Git’s cat-file tool, which can be used to print out the contents of arbitrary objects.

    Back when v2.38.0 was released, we talked about how cat-file gained support to apply Git’s mailmap rules when printing out the contents of a commit. To summarize, Git allows rewriting name and email pairs according to a repository’s mailmap. In v2.38.0, git cat-file learned how to apply those transformations before printing out object contents with the new --use-mailmap option.

    But what if you don’t care about the contents of a particular object, and instead want to know the size? For that, you might turn to something like --batch-check=%(objectsize), or -s if you’re just checking a single object.

    But you’d be mistaken! In previous versions of Git, both the --batch-check and -s options to git cat-file ignored the presence of --use-mailmap, leading to potentially incorrect results when the name/email pairs on either side of a mailmap rewrite were different lengths.

    In Git 2.40, this has been corrected, and git cat-file -s and --batch-check with will faithfully report the object size as if it had been written using the replacement identities when invoked with --use-mailmap.

    [source]

  • While we’re talking about scripting, here’s a lesser-known Git command that you might not have used: git check-attr. check-attr is used to determine which gitattributes are set for a given path.

    These attributes are defined and set by one or more .gitattributes file(s) in your repository. For simple examples, it’s easy enough to read them off from a .gitattributes file, like this:

    $ head -n 2 .gitattributes 
    * whitespace=!indent,trail,space 
    *.[ch] whitespace=indent,trail,space diff=cpp
    

    Here, it’s relatively easy to see that any file ending in *.c or *.h will have the attributes set above. But what happens when there are more complex rules at play, or your project is using multiple .gitattributes files? For those tasks, we can use check-attr:

    $ git check-attr -a git.c 
    git.c: diff: cpp 
    git.c: whitespace: indent,trail,space
    

    In the past, one crucial limitation of check-attr is that it required an index, meaning that if you wanted to use check-attr in a bare repository, you had to resort to temporarily reading in the index, like so:

    TEMP_INDEX="$(mktemp ...)" 
    
    git read-tree --index-output="$TEMP_INDEX" HEAD 
    GIT_INDEX_FILE="$TEMP_INDEX" git check-attr ... 
    

    This kind of workaround is no longer required in Git 2.40 and newer. In Git 2.40, check-attr supports a new --source= to scan for .gitattributes in, meaning that the following will work as an alternative to the above, even in a bare repository:

    $ git check-attr -a --source=HEAD^{tree} git.c 
    git.c: diff: cpp 
    git.c: whitespace: indent,trail,space
    

    [source]

  • Over the years, there has been a long-running effort to rewrite old parts of Git from their original Perl or Shell implementations into more modern C equivalents. Aside from being able to use Git’s own APIs natively, consolidating Git commands into a single process means that they are able to run much more quickly on platforms that have a high process start-up cost, such as Windows.

    On that front, there are a couple of highlights worth mentioning in this release:

    In Git 2.40, git bisect is now fully implemented in C as a native builtin. This is the result of years of effort from many Git contributors, including a large handful of Google Summer of Code students.
    Similarly, Git 2.40 retired the legacy implementation of git add --interactive, which also began as a Shell script and was re-introduced as a native builtin back in version 2.26, supporting both the new and old implementation behind an experimental add.interactive.useBuiltin configuration.

    Since that default has been “true” since version 2.37, the Git project has decided that it is time to get rid of the now-legacy implementation entirely, marking the end of another years-long effort to improve Git’s performance and reduce the footprint of legacy scripts.

    [source, source]

  • Last but not least, there are a few under-the-hood improvements to Git’s CI infrastructure. Git has a handful of long-running Windows-specific CI builds that have been disabled in this release (outside of the git-for-windows repository). If you’re a Git developer, this means that your CI runs should complete more quickly, and consume fewer resources per push.

    On a similar front, you can now configure whether or not pushes to branches that already have active CI jobs running should cancel those jobs or not. This may be useful when pushing to the same branch multiple times while working on a topic.

    This can be configured using Git’s ci-config mechanism, by adding a special script called skip-concurrent to a branch called ci-config. If your fork of Git has that branch then Git will consult the relevant scripts there to determine whether CI should be run concurrently or not based on which branch you’re working on.

    [source, source]

  The rest of the iceberg

  That’s just a sample of changes from the latest release. For more, check out the release notes for 2.40, or any previous version in the Git repository.

  
  
  https://github.blog/2023-03-13-highlights-from-git-2-40/
• 

  How GitHub Docs’ new search works
  • sorangutan  

  1
  1
  Posts
  23
  Views

  

  Until recently, the site-search on GitHub Docs was an in-memory solution. While it was a great starting point, we ultimately needed a solution that would scale with our growing needs, so we rewrote it in Elasticsearch. In this blog post, we share how the implementation works and how you can impress users with your site-search by doing the same.

  How it started

  Our previous solution couldn’t scale because it required loading all of the records into memory (the Node.js code that Express.js runs). This means that we’d need to scrape all the searchable text—title, headings, breadcrumbs, content, etc.—and store that data somewhere so it could quickly be loaded in the Node process runtime. To store the data, we used the Git repository itself, so when we built the Docker image run in Azure, it would have access to all the searchable text from disk.

  It would take a little while to load in all the searchable text, so we generated a serialized index from the searchable text, and stored that on disk too. This solution is OKish if your data is small, but we have eight languages and five different versions per language.

  Running Elasticsearch locally

  The reason we picked Elasticsearch over other alternatives is a story for another day, but one compelling argument is that it’s possible to run it locally on your laptop. For the majority of contributors who make copy edits, the task of installing Elasticsearch locally isn’t necessary. So by default, our /api/search Express.js middleware looks something like this:

  if (process.env.ELASTICSEARCH_URL) {
    router.use('/search', search)
  } else {
    router.use(
      '/search',
      createProxyMiddleware({
        target: 'https://docs.github.com',
    ...
  

  When someone uses http://localhost:4000/api/search on their laptop (or Codespaces) it simply forwards the Elasticsearch stuff to our production server. That way, engineers (like myself) who are debugging the search engine locally can start an Elasticsearch on http://localhost:9200 and set that in their .env file. Now engineers can quickly try new search query techniques entirely with their own local Elasticsearch.

  The search implementation

  A core idea in our search implementation is that we make only one single query to Elasticsearch, which contains our entire specification for how we want results ranked. Rather than sending a single request, we could try a more specific search query first, then, if there are too few results, we could attempt a second, less defined search query:

  // NOTE! This is NOT what we do.
  
  let result = await client.search({ index, body: searchQueryStrict })
  if (result.hits.length === 0) {
    // nothing found when being strict, try again with a loose query
    result = await client.search({ index, body: searchQueryLoose })
  }
  

  To ensure we get the most exact results, we use boosts and a matrix of various matching techniques. In somewhat pseudo code it looks like this:

  If the query is multiple terms (we’ll explain what _explicit means later):

  [
    { match_phrase: { title_explicit: [Object] } },
    { match_phrase: { title: [Object] } },
    { match_phrase: { headings_explicit: [Object] } },
    { match_phrase: { headings: [Object] } },
    { match_phrase: { content: [Object] } },
    { match_phrase: { content_explicit: [Object] } },
    { match: { title_explicit: [Object] } },
    { match: { headings_explicit: [Object] } },
    { match: { content_explicit: [Object] } },
    { match: { title: [Object] } },
    { match: { headings: [Object] } },
    { match: { content: [Object] } },
    { match: { title_explicit: [Object] } },
    { match: { headings_explicit: [Object] } },
    { match: { content_explicit: [Object] } },
    { match: { title: [Object] } },
    { match: { headings: [Object] } },
    { match: { content: [Object] } },
    { fuzzy: { title: [Object] } }
  ]
  

  If the query is a single term:

  [
    { match: { title_explicit: [Object] } },
    { match: { headings_explicit: [Object] } },
    { match: { content_explicit: [Object] } },
    { match: { title: [Object] } },
    { match: { headings: [Object] } },
    { match: { content: [Object] } },
    { fuzzy: { title: [Object] } }
  ]
  

  Sure, it can look like a handful of queries, but Elasticsearch is fast. Most of the total time is the network time to send the query and receive the results. In fact, the time it takes to execute the entire search query (excluding the networking) hovers quite steadily at 20 milliseconds on our Elasticsearch server.

  

  About 55% of all searches on docs.github.com are multi-term queries, for example, actions rest. Meaning that in roughly half the cases, we can use the simplified queries because we can omit things like the match_phrase parts of the total query.

  Before getting into the various combinations of searches, there’s a section later about what “explicit” means. Essentially, each field is indexed twice. E.g. title and title_explicit. It’s the same content underneath but it’s tokenized differently, which affects how it matches to queries—and that difference is exploited by having different boostings.

  The nodes that make up the matrix are:

  Fields:

  • title (the <h1> text)
  • headings (the <h2> texts)
  • content (the bulk of the article text)

  Analyzer:

  • explicit (no stemming and no synonyms)
  • regular (full Snowball stemming and possibly synonyms)

  Matches: (on multi-term queries)

  • match_phrase
  • match with OR (docs that contain “foo” OR “bar”)
  • match with AND (docs that contain “foo” AND “bar”)

  Each of these combinations has a unique boost number, which boosts the ranking of the matched result. The actual number doesn’t matter much, but what matters is that the boost numbers are different from each other. For example, a match on the title has a slightly higher boost than a match on the content. And a match where all words are present has a slightly higher boost than when only some of the words match. Another example: if the search term is docker action, the users would prefer to see “Creating a Docker container action” ahead of “Publishing Docker images” or “Metadata syntax for GitHub Actions.”

  Each of the above nodes has a boost calculation that looks something like this:

  const BOOST_PHRASE = 10.0
  const BOOST_TITLE = 4.0
  const BOOST_HEADINGS = 3.0
  const BOOST_CONTENT = 1.0
  const BOOST_AND = 2.5
  const BOOST_EXPLICIT = 3.5
  
  ...
  match_phrase: { title_explicit: { boost: BOOST_EXPLICIT * BOOST_PHRASE * BOOST_TITLE, query } },
  match: { headings: { boost: BOOST_HEADINGS * BOOST_AND, query, operator: 'AND' } },
  ...
  

  If you exclusively print out what the boost value becomes for each node in the matrix you get:

  [
    { match_phrase: { title_explicit: 140 } },
    { match_phrase: { title: 40 } },
    { match_phrase: { headings_explicit: 105 } },
    { match_phrase: { headings: 30 } },
    { match_phrase: { content: 10 } },
    { match_phrase: { content_explicit: 35 } },
    { match: { title_explicit: 35, operator: 'AND' } },
    { match: { headings_explicit: 26.25, operator: 'AND' } },
    { match: { content_explicit: 8.75, operator: 'AND' } },
    { match: { title: 10, operator: 'AND' } },
    { match: { headings: 7.5, operator: 'AND' } },
    { match: { content: 2.5, operator: 'AND' } },
    { match: { title_explicit: 14 } },
    { match: { headings_explicit: 10.5 } },
    { match: { content_explicit: 3.5 } },
    { match: { title: 4 } },
    { match: { headings: 3 } },
    { match: { content: 1 } },
    { fuzzy: { title: 0.1 } }
  ]
  

  So that’s what we send to Elasticsearch. It’s like a complex wishlist saying, “I want a bicycle for Christmas. But if you have a pink one, even better. And a pink one with blue stripes is even better still. Actually, bestest would be a pink one with blue stripes and a brass bell.”

  What’s important, in terms of an ideal implementation and search result for users, is that we use our human and contextual intelligence to define these parameters. Some of it is fairly obvious and some is more subtle. For example, we think a phrase match on the title without the need for stemming is the best possible match, so that gets the highest boost.

  Why explicit boost is important

  If someone types “creating repositories” it should definitely match a title like “Create a private GitHub repository” because of the stems creating => creat <= create and repositories => repository <= repository. We should definitely include those matches that take stemming into account. But if there’s an article that explicitly uses the words that match what the user typed, like “Creating private GitHub repositories,” then we want to boost that article’s ranking because we think that’s more relevant to the searcher.

  Another good example of this is the special keyword working-directory which is an actual exact term that can appear inside the content. If someone searches for working-directory, we don’t want to let an (example) title like “Directories that work” overpower the rankings when working-directory and “Directories that work” are both deconstructed to the same two stems [ 'work', 'directori' ].

  The solution we rely on is to make two matches: one with stemming and one without. Each one has a different boost. It’s similar to saying, “I’m looking for a ‘Peter’ but if there’s a ‘Petter’ or ‘Pierre’ or ‘Piotr’ that’ll also do. But ideally, ‘Peter’ first and foremost. In fact, give me all the results, but ‘Peter’ first.” So this is what we do. The stemming is great but it can potentially “overpower” search results. This helps with specific keywords that might appear to be English prose. For example “working-directory” even looks like a regular English expression but it’s actually a hardcoded specific keyword.

  In terms of code, it looks like this:

  // Creating the index...
  await client.indices.create({
    mappings: {
      properties: {
        url: { type: 'keyword' },
        title: { type: 'text', analyzer: 'text_analyzer', norms: false },
        title_explicit: { type: 'text', analyzer: 'text_analyzer_explicit', norms: false },
        content: { type: 'text', analyzer: 'text_analyzer' },
        content_explicit: { type: 'text', analyzer: 'text_analyzer_explicit' },
        // ...snip...
        },
      },
      // ...snip...
    })
  // Searching...
  matchQueries.push(
    ...[
      { match: { title_explicit: { boost: BOOST_EXPLICIT * BOOST_TITLE, query } } },
      { match: { content_explicit: { boost: BOOST_EXPLICIT * BOOST_CONTENT, query } } },
      { match: { title: { boost: BOOST_TITLE, query } } },
      { match: { content: { boost: BOOST_CONTENT, query } } },
      // ...snip...
  ])
  

  Ranking is not that easy

  Search is quickly becoming an art. It’s not enough to just match the terms and display a list of documents that match the terms of the input. For starters, the ranking is crucially important. This is especially true when a search term yields tens or hundreds of matching documents. Years of depending on Google has taught us all to expect the first search result to be the one we want.

  To make this a great experience, we try to infer which document the user is genuinely looking for by using pageview metrics as a way to determine which page is most popular. Sure, if the most popular one is offered first, and gets the clicks, it’ll just get even more popular, but it’s a start. We get a lot of pageviews from users Googling something. But we also get regular pageview metrics from users simply navigating themselves to the pages they have figured out has the best information for them.

  At the moment, we gather pageview metrics for the top 1,000 most popular URLs. Then we rank them and normalize it to a number between (and including) 0.0 to 1.0. Whatever the number is we add +1.0 and then we multiply this number in Elasticsearch with the match score.

  Suppose a search query finds two documents that match, and their match score is 15.6 and 13.2 based on the search implementation mentioned above. Now, suppose that match of 13.2 is on a popular page, its popularity number might be 0.75, so it becomes 13.2 * (1 + 0.75) = 23.1. And the other one that matched a little bit better, has a popularity number of 0.44, so its final number becomes 15.6 * (1 + 0.44) = 22.5, which is less. In conclusion, it gives documents that might not be term-for-term as “matchy” as others a chance to rise above. This also ensures that a hugely popular document that only matched vaguely in the content won’t “overpower” other matches that might have matched in the title.

  It’s a tricky challenge, but that’s what makes it so much fun. You have to bake in some human touch to the code as a way of trying to think like your users. It’s also an algorithm that will never reach perfection because even with more and better metrics, the landscape of users and where they come from, is constantly changing. But we’ll try to keep up.

  What’s next?

  Elasticsearch has a functionality where you can define aliases for words, which they call synonyms. (For example, repo = repository.) The challenge with this feature is how it’s managed and maintained by writers in a convenient and maintainable way.

  Currently at GitHub, we base our popularity numbers by pageview metrics. It would be interesting to dig deeper into which page the user lands on. As an example, suppose the reader isn’t sure how to find what they need, so they start on a product landing page (we have about 20 of those) and slowly make their way deeper into the content and finally arrive on the page that supplied the knowledge or answer they want. This process would give each page an equal measure, and that’s not great.

  Another exciting idea is to record all the times a search result URL is not clicked on, especially when it shows up higher in the ranking. That would decrease the risk of popular listings only getting more popular. If you record when something is “corrected” by humans, that could be a very powerful signal.

  You could also tailor the underlying search based on other contextual variables. For example, if a user is currently inside the REST API docs you could infer that REST API-related docs are slightly preferred when searching for something ambiguous like “billing” (i.e. prefer REST API “About billing” not Billing and payments “Setting your billing email.”).

  What are you looking for next? Have you tried the search on https://docs.github.com recently and found that it wasn’t giving you the best search result? Please let us know and get in touch.

  
  
  https://github.blog/2023-03-09-how-github-docs-new-search-works/
• 

  How to automate your dev environment with dev containers and GitHub Codespaces
  • sorangutan  

  1
  1
  Posts
  29
  Views

  

  When I started my first role as a software engineer, I remember taking about four days to set up my local development environment. I had so many issues with missing dependencies, incorrect versions, and failed installations. When I finally finished setting up all the tools and software I needed to be a productive member of the team, I cloned one of our repositories to my machine, set up my environment variables, ran npm run dev and received so many errors because I forgot to install the dependencies (and read the README) or switch to the right node version. Ugh! I can’t tell you how many times this happened to me in my first year!

  Back then, I wished I had a way that was streamlined—something I set up only once, that just worked every time I accessed the repository. Although I did learn how to automate my computer setup with a Brewfile, I wish I could just get to coding in a repository without thinking about configuration.

  
  source

  When I think about how we work on projects in a repository, I realize that many of the processes we need to get started on that project can be automated with the help of dev containers, in this case, by using a devcontainer.json file and Codespaces.

  Let’s take a look at how we can automate our dev environment by adding a dev container to this open source project—Tech is Hiring in GitHub Codespaces.

  For a TLDR of this post, GitHub Codespaces enables you to start coding faster when coupled with dev containers. See image below for a summary of how:

  

  Now, let’s get some definitions out of the way.

  What is GitHub Codespaces?

  GitHub Codespaces is a development environment in the cloud. It is hosted by GitHub in an isolated environment (Docker container) that runs on a virtual machine. If you’re not familiar with virtual machines or Docker containers, take a look at these videos: what is a virtual machine? and what are Docker containers?.

  Currently, individual developers have 60 hours of free codespaces usage per month, so definitely take advantage of this awesomeness to build from anywhere.

  What are dev containers?

  Dev containers allow us to run our code in a preconfigured, isolated environment (container). It gives us the ability to predefine our dev environment in our repositories and use a consistent, reliable environment across the board without worrying about configuration files—since it’s all set up for us from the beginning with a devcontainer.json file.

  What is the devcontainer.json file?

  The devcontainer.json file is a document that defines how your project will build, display, and run. Simply put, it allows us to specify the behavior we want in our dev environment. For example, if we have a devcontainer.json file that includes installing the ESLint extension in VS Code, once we open up a workspace, ESLint will be automatically installed for us.

  Automating your workflow with dev containers and GitHub Codespaces

  To start using GitHub Codespaces, we don’t need to set up a devcontainer.json file. By default, GitHub Codespaces uses the universal dev container image, which caters to a vast array of languages and tools. This means, whenever you open up a new codespace without a devcontainer.json file your codespace will automagically load so you can code instantly. However, adding a devcontainer.json file gives us the ability to automate a lot of our dev environment workflows to our liking.

  Okay, okay, that was a lot of chatter—let’s now get into what you really came here for!

  
  source

  Using the open source project, Tech is Hiring, let’s walk through how we typically work with a repository using our local dev environment.

  What work typically looks like

  At first glance, we see that this project uses Nextjs, Tailwind CSS, Chakra UI, TypeScript, Storybook, Vite, Cypress, Axios, and Reactjs as some of its dependencies. We’d need to install all these dependencies to our local machine to get this project running.

  1. Let’s clone the repository, and cd into the project.

     

  2. Then, let’s install dependencies to get the project running locally.

     

  3. This project uses storybook, so let’s run both storybook and spin up the actual app locally.

     

  The process is not so bad, but it took a bit of time. We also need to check to make sure we’re using the correct node version, check if we need any environment variables, and if there are any runtime errors to resolve. Thankfully, I didn’t encounter any errors while working on this, but it still took a bit of time.

  Going faster with GitHub Codespaces and dev containers

  Let’s make the process better by adding a devcontainer.json file to this project and opening it in GitHub Codespaces to see what happens.

  We can either use the VS Code command palette to add a pre-existing dev container or we can write the configuration file ourselves (which we’ll do below).

  1. Let’s first create a .devcontainer folder in the root of the project and a devcontainer.json file in the new folder.

     

  2. Now, let’s automate installing dependencies, starting the dev server, opening a preview of our app on localhost:3000, and installing vscode extensions. Once we get everything configured, your json file should look like this:

     {
       // image being used
        "image": "mcr.microsoft.com/devcontainers/universal:2",
       // set minimum cpu
        "hostRequirements": {
            "cpus": 4
        },
        // install dependencies and start app
        "updateContentCommand": "npm install",
        "postAttachCommand": "npm run dev",
        // open app.tsx once container is built
        "customizations": {
            "codespaces": {
                "openFiles": [
                    "src/pages/_app.tsx"
                ]
            },
            // install some vscode extensions
            "vscode": {
                "extensions": [
                    "dbaeumer.vscode-eslint",
                    "github.vscode-pull-request-github",
                    "eamodio.gitlens",
                    "christian-kohler.npm-intellisense"
                ]
            }
        },
        // connect to remote server
        "forwardPorts": [3000],
        // give port a label and open a preview of the app
        "portsAttributes": {
           "3000": {
              "label": "Application",
              "onAutoForward": "openPreview"
            }
          }
     }
     

     Sidenote: I’ve broken down the purpose of the properties in this file for you by adding comments before each. To learn more about each property, continue reading at container.dev. I also installed a few extensions that are not needed, but I wanted to show you that you could automate installing extensions, too!

  3. Let’s commit the file and merge it into the main branch, then open up the project in GitHub Codespaces.

     

     When we open up the application in GitHub Codespaces, our dependencies will be installed, the server will start, and a preview will automatically open for us. If we needed environment variables to work on this repository, those would have already been configured for us as a repository secret on GitHub. We also didn’t need to install hefty node_modules to our machine.

  I call this a win!

  Comparing both ways

  There’s plenty more that we can do with dev containers and GitHub Codespaces to automate our dev environment. But let’s summarize what we just did in GitHub Codespaces and with the help of dev containers:

  • We clicked the GitHub Codespaces button on the GitHub repository.
  • Everything was setup/installed for us (thanks json file!).
  • We got to work and started coding.

  Now, isn’t that better?

  Wrapping up

  So, what’s the point of GitHub Codespaces and why should you care as a developer? Well, for one, you can automate most of the startup processes you need to access a repository. You can also do a lot more customizations to your dev environment with dev containers. Take a look at all the options you have—and watch out for my next blog post where I’ll go through the anatomy of a devcontainer.json file.

  Secondly, you can code from anywhere. I hate it when I’m not able to access one of my side projects on a different machine because that one machine is configured perfectly to suit the project. With GitHub Codespaces, you can start coding at the click of a button and from any machine that supports a modern browser.

  I encourage you to get started with GitHub Codespaces today and try adding a devcontainer.json file to one of your projects! I promise you won’t regret it.

  Until next time, happy coding!

  
  
  https://github.blog/2023-03-06-how-to-automate-your-dev-environment-with-dev-containers-and-github-codespaces/
• 

  Why Python keeps growing, explained
  • sorangutan  

  1
  1
  Posts
  43
  Views

  

  Which programming language has been around for more than three decades and continues to grow in popularity each year?

  If you guessed Python, you nailed it. In the 2022 Octoverse report, we found that Python remains the second most-used programming language on GitHub. Interestingly, Python’s use grew more than 22 percent year over year with more than four million developers on GitHub using it at some point in 2022.

  In this article, we’ll dive into a brief history of Python, its benefits, its use cases, and seek to answer why a program language conceived in the 1980s continues to dominate development. And, since this is GitHub, we’ll also offer a few useful tips and tricks for developers new to—and experienced in—Python.

  So, what is Python?

  Python is a high-level, interpreted programming language with a simple syntax, which makes it easily readable and extremely user- and beginner-friendly. Originally built to satisfy Guido Van Rossum’s desire for a programming language that was simple to use and beautiful to look at, Python was first released to the world in 1991.

  Fun fact: Python was named after the BBC TV show, “Monty Python’s Flying Circus.”

  Since its development, it has grown to have widespread applicability for developers, data scientists, researchers, and more. But how, you may ask, can a coding language be simple and beautiful to look at? Here’s some proof:

  Python

  print("Hello world.")
  

  vs.

  Java

  public class HelloWorld {
      public static void main (String[]args) {
        System.out.println.("Hello world");
      }
  }
  

  Since Python is a general-purpose language, it can be used in a variety of applications, and its uncomplicated nature makes it an excellent language for automating tasks, building websites or software, and analyzing data.

  Python also has several other characteristics that make it popular amongst developers and engineers. These include:

  • It’s easy to read. Python code uses English keywords rather than punctuation, and its line breaks help define the code blocks. In practice, this means you can identify what the code is designed to do simply by looking at it.

  • It’s open source. You can download the source code, modify it, and use it however you want.

  • It’s portable. Some languages require you to modify code to run on different platforms, but Python is a cross-platform language, which means you can run the same code on any operating system with a Python interpreter.

  • It’s extendable. Python code can be written in other languages (such as C++), and users can add low-level modules to the Python interpreter to customize and optimize their tools.

  • It has a broad standard library. This library is available for anyone to access and means that users don’t have to write code for every single function—they can access built-in modules that help with issues in everyday programming and more.

  What is Python commonly used for?

  Python can be used for just about anything, from web and software development to machine learning and artificial intelligence (AI). Let’s take a look at some of its most common use cases.

  import antigravity
  
  def main():
      antigravity.fly()
  
  if __name__ == '__main__':
      main()
  

  Run this command to check out an inside joke among Python developers.

  Using Python for web and software development

  Python is a popular language for web and software development because you can create complex, multi-protocol applications while maintaining concise, readable syntax. In fact, some of the most popular applications were built with Python. Plus, Python’s open source community provides developers with an extensive amount of reusable code, frameworks, and support. Case in point: Django is one of the most-used Python frameworks designed by experienced developers to help others accelerate their application build times and avoid issues that might balk their progress.

  Using Python for task automation

  One of Python’s key benefits is its ability to automate manual, repetitive tasks. With Python, you can learn how to automate just about anything by using either built-in modules or pre-written code from its robust library. Or you can write your own custom scripts to perform specific actions. For example, you can easily automate emails with the “smtplib” module or copy files with the “shutil” module. Python also has a robust set of testing frameworks, which makes it an excellent language for test automation. Frameworks such as Pytest, Behave, and Robot allow developers to write simple yet effective tests to ensure the quality of their builds.

  Using Python for machine learning and data science

  Here’s a fun fact: Python is the top preferred language for data science and research. Since its syntax is easily understandable and adaptable, people with little-to-no development experience can easily learn Python and use it to manipulate data for research, reporting, predictable or regression analyses, and more. Collecting and parsing data can be a time-consuming task for data scientists. Python is also one of the top languages for training machine learning (ML) models. Through specific algorithms, these models can analyze and identify patterns in data to make predictions or decisions based on that data. They also constantly evolve based on outputs of previous datasets to confront new variables. Data scientists and developers training ML models often utilize libraries, such as NumPy, Pandas, and Matplotlib, to automate functions like cleaning, data transformation, and visualization.

  Using Python for financial analysis

  Similar to how Python can assist data scientists with the heavy lift of large data sets, Python is widely used in the financial industry to quickly perform complex computations. Stock markets generate huge amounts of data, and Python can be used to import data on stock prices and generate strategies through algorithms to identify trading opportunities. The language can also be used for portfolio optimization, risk management, financial modeling and visualization, cryptocurrency analysis, and even fraud detection.

  Using Python for and artificial intelligence

  Python can also be found in some of the most complex, artificial intelligence (AI) technologies—and it’s actually one of the preferred languages for AI. Python’s concise and readable code allows developers to create consistent, reliable systems, and its vast library provides a number of frameworks like PyBrain, which offers developers powerful algorithms for machine learning tasks. Plus, Python’s visualization capabilities can help convert these large datasets for AI or ML into comprehensible graphs or reports. Interestingly enough, OpenAI, the artificial intelligence research lab, utilizes the Python framework, Pytorch, as their standard framework for deep learning, which trains its AI systems.

  Why is Python so popular?

  In addition to its relative simplicity to learn, there are a few other reasons why Python continues to consistently grow in popularity. These include:

  • It’s more productive. Compared to some other more complex programming languages like C++, Python’s syntax allows users to do more with less and cut down on time and effort to write the same lines of code.

  • It has an expansive, supportive community of users. Even the best developers run into problems— and this is where user communities can become an invaluable resource. Python has a huge community with documentation, tutorials, tips, and tricks to master the language. The Python community on GitHub, for example, offers everything from information on the latest version of the language to bug reports and update notes.

  • It’s academic. Python has become the go-to language in academia with some students even encountering Python as early as elementary school. (Believe it or not, there are children’s picture books dedicated to Python.) While computer science students are often taught Python, its use extends beyond that discipline into other areas of STEM and academic research. For example, Python can be used to solve differential equations, perform statistical analyses, simulate and track particle diffusion, and more.

  • It has high corporate demand. Because of its wide scale applicability in development and data analysis work, learning and knowing Python is often considered a top-skill among job seekers. According to Statista, Python was the third most demanded language in 2022 by recruiters worldwide.

  The bottom line

  Python is everywhere—and it’s been used to build a significant number of the technologies, websites, and even systems most people encounter on a daily basis. It powers everything from your favorite video streaming service to the ML algorithms that can help you make your next cryptocurrency trade. And for an even broader scope example (pun absolutely intended), NASA uses Python to power data analysis with its sophisticated James Webb Space Telescope, which makes it one of the few programming languages that is, quite literally, out of this world.

  How to get started with Python

  A quick Google search will yield hundreds of resources out there to jumpstart your Python journey—and that can quickly get a little overwhelming. To simplify things, here are a few helpful GitHub repositories to help you get started with Python:

  • Explore pre-built Python algorithms: From networking flows to physics and neural networks, this repository is a great guide to building algorithms in Python.
  • Learn Python in 30 days: This step-by-step guide will walk you through the basics of Python in 30 days.
  • Grab some tips from this cheatsheet: Check out this collection of Python scripts with code examples and explanations for learning the language.
  • Sharpen your Python skills: Pick up some tips from this study guide for both beginner and seasoned users built by a Python superfan!

  To get started, download the latest version of Python.

  Start building on GitHub today

  GitHub offers two easier ways to start working with Python: GitHub Codespaces and GitHub Copilot.

  You can start building today for free with GitHub Codespaces, which every developer on GitHub gets 60 free hours of use time per month to spin up a development environment in the cloud from any device at speed. Check out the Django quick start template to begin coding right in your browser!

  You can also use GitHub Copilot, GitHub’s AI pair programmer, to write your first lines of Python. Here’s how:

  1. Install the GitHub Copilot extension into your code editor.
  2. Describe the purpose of your project in a comment.
  3. Write a comment describing which libraries you may need.
  4. Start tabbing and let GitHub Copilot suggest lines of code to help you learn new techniques or methods.

  From machine learning to data analysis, Python’s versatility allows it to continue its explosive growth with developers and non-developers alike. Experiment with Python through GitHub or on your local machine to be part of this growth and get started today!

  
  
  https://github.blog/2023-03-02-why-python-keeps-growing-explained/
• 

  GitHub Availability Report: February 2023
  • sorangutan  

  1
  1
  Posts
  18
  Views

  

  In February, we experienced three incidents that resulted in degraded performance across GitHub services. This report also sheds light into a January incident that resulted in degraded performance for GitHub Packages and GitHub Pages and another January incident that impacted Git users.

  January 30 21:31 UTC (lasting 35 minutes)

  On January 30 at 21:36 UTC, our alerting system detected a 500 error response increase in requests made to the Container registry. As a result, most builds on GitHub Pages and requests to GitHub Packages failed during the incident.

  Upon investigation, we found that a change was made to the Container registry Redis configuration at 21:30 UTC to enforce authentication on Redis connections. There was an issue with the Container registry production deployment file where client connections were unable to authenticate due to a hard coded connection string, resulting in errors and preventing successful connections.

  At 22:12 UTC, we reverted the configuration change for Redis authentication. Container registry began recovering two minutes later, and GitHub Pages was considered healthy again by 22:21 UTC.

  To help prevent future incidents, we improved management of secrets in the Container registry’s Redis deployment configurations and added extra test coverage for authenticated Redis connections.

  January 30 18:35 UTC (lasting 7 hours)

  On January 30 at 18:35 UTC, GitHub deployed a change which slightly altered the compression settings on source code downloads. This change altered the checksums of the resulting archive files, resulting in unforeseen consequences for a number of communities. The contents of these files were unchanged, but many communities had come to rely on the precise layout of bytes also being unchanged. When we realized the impact we reverted the change and communicated with affected communities.

  We did not anticipate the broad impact this change would have on a number of communities and are implementing new procedures to prevent future incidents. This includes working through several improvements in our deployment of Git throughout GitHub and adding a checksum validation to our workflow.

  See this related blog post for details about our plan going forward.

  February 7 21:30 UTC (lasting 20 hours and 35 minutes)

  On February 7 at 21:30 UTC, our monitors detected failures creating, starting, and connecting to GitHub Codespaces in the Southeast Asia region, caused by a datacenter outage of our cloud provider. To reduce the impact to our customers during this time, we redirected codespace creations to a secondary location, allowing new codespaces to be used. Codespaces in that region recovered automatically when the datacenter recovered, allowing existing codespaces in the region to be restarted. Codespaces in other regions were not impacted during this incident.

  Based on learnings from this incident, we are evaluating expanding our regional redundancy and have started making architectural changes to better handle temporary regional and datacenter outages, including more regularly exercising our failover capabilities.

  February 18 02:36 UTC (lasting 2 hours and 26 minutes)

  On February 18 at 02:36 UTC, we became aware of errors in our application code pointing to connectivity issues to our MySQL databases. Upon investigation, we believe these errors were due to a few unhealthy deployments of our sharding middleware. At 03:30 UTC, we performed a re-deployment of the database infrastructure in an effort to remediate. Unfortunately, this propagated the issue to all Kubernetes pods, leading to system-wide errors. As a result, multiple services returned 500 error responses and GitHub users were experiencing issues signing in to GitHub.com.

  At 04:30 UTC, we found that the database topology in 30% of our deployments was corrupted, which prevented applications from connecting to the database. We applied a copy of the correct database topology to all deployments, which resolved the errors across services by 05:00 UTC. Users were then able to sign in to GitHub.com.

  To help prevent future incidents, we added a monitor to detect database topology errors so we can identify this well in advance of these changes impacting production systems. We have also improved our observability around topology reloads, both successful and erroneous ones. We are also doing a deeper review of the contributing factors to this incident to learn and improve both our architecture and operations to prevent a recurrence.

  February 28 16:05 UTC (lasting 1 hour and 26 minutes)

  On February 28 at 16:05 UTC, we were notified of degraded performance for GitHub Codespaces. We resolved the incident at 17:31 UTC.

  Due to the recency of this incident, we are still investigating the contributing factors and will provide a more detailed update in next month’s report.

  ---

  Please follow our status page for real-time updates on status changes. To learn more about what we’re working on, check out the GitHub Engineering Blog.

  
  
  https://github.blog/2023-03-01-github-availability-report-february-2023/
• 

  GitHub Actions: Introducing faster GitHub-hosted x64 macOS runners
  • sorangutan  

  1
  1
  Posts
  19
  Views

  

  Today, GitHub is releasing a public beta for all new, more powerful hosted macOS runners for GitHub Actions. Teams who are looking to speed up their macOS jobs now have new options to meet their needs.

  Faster GitHub-hosted macOS runners

  When developers use GitHub-hosted runners for GitHub Actions, GitHub is always working to give teams performance and flexibility to quickly build for different platforms. The new GitHub-hosted macOS runners are based on the most current x64 Apple hardware and reduce the time it takes to run your jobs. This means less time waiting on your CI and more time building and rapidly testing code.

  In addition to the existing 3-core macOS hosted runners, we are also introducing a new “XL” size, 12-core macOS runner to allow you to pay for that extra performance when you need it. Increasing your runner size is as simple as changing a single line in your workflow file, making it super easy to speed up lagging builds, and with on-demand availability when you want it, you only pay for what you use.

  PyTorch, the open source machine learning framework, has used the new runner to improve build times from 1.5 hours to 30 minutes!

  If you’re already using the 3-core macOS runners, simply request to join the beta, and we’ll give you access to the hardware to our new fleet. To try the new 12-core runner runner, update the runs-on: key in your GitHub Actions workflow YAML to target the macos-12-xl or macos-latest-xl runner images once you’re in the beta.

  

  The new macOS XL runner is always billed in both private and public repositories and so does not consume any of the free included GitHub Action minutes. To learn more about runner per job minute pricing, check out the docs.

  Learn more and join the beta

  To learn more about using the new macOS runner, check out the hosted runner docs.

  Sign up for the beta today.

  Apple silicon is on the way!

  We’ve started racking up and testing out Apple silicon-based machines in the hosted runner data centers. Follow this roadmap item to track progress. We hope to be offering a limited beta in CY2023 depending on hardware availability, but keep your eyes on the GitHub blog for more information.

  
  
  https://github.blog/2023-03-01-github-actions-introducing-faster-github-hosted-x64-macos-runners/
• 

  How to build a consistent workflow for development and operations teams
  • sorangutan  

  1
  1
  Posts
  20
  Views

  

  In GitHub’s recent 2022 State of the Octoverse report, HashiCorp Configuration Language (HCL) was the fastest growing programming language on GitHub. HashiCorp is a leading provider of Infrastructure as Code (IaC) automation for cloud computing. HCL is HashiCorp’s configuration language used with tools like Terraform and Vault to deliver IaC capabilities in a human-readable configuration file across multi-cloud and on-premises environments.

  HCL’s growth shows the importance of bringing together the worlds of infrastructure, operations, and developers. This was always the goal of DevOps. But in reality, these worlds remain siloed for many enterprises.

  In this post we’ll look at the business and cultural influences that bring development and operations together, as well as security, governance, and networking teams. Then, we’ll explore how GitHub and HashiCorp can enable consistent workflows and guardrails throughout the entire CI/CD pipeline.

  The traditional world of operations (Ops)

  Armon Dadgar, co-founder of HashiCorp, uses the analogy of a tree to explain the traditional world of Ops. The trunk includes all of the shared and consistent services you need in an enterprise to get stuff done. Think of things like security requirements, Active Directory, and networking configurations. A branch represents the different lines of business within an enterprise, providing services and products internally or externally. The leaves represent the different environments and technologies where your software or services are deployed: cloud, on-premises, and container environment, among others.

  In many enterprises, the communication channels and processes between these different business areas can be cumbersome and expensive. If there is a significant change to the infrastructure or architecture, multiple tickets are typically submitted to multiple teams for reviews and approvals across different parts of the enterprise. Change Advisory Boards are commonly used to protect the organization. The change is usually unable to proceed unless the documentation is complete. Commonly, there’s a set of governance logs and auditable artifacts which are required for future audits.

  Wouldn’t it be more beneficial for companies if teams had an optimized, automated workflow that could be used to speed up delivery and empower teams to get the work done in a set of secure guardrails? This could result in significant time and cost savings, leading to added business value.

  After all, a recent Forrester report found that over three years, using GitHub drove 433% ROI for a composite organization simply with the combined power of all GitHub’s enterprise products. Not to mention the potential for time savings and efficiency increase, along with other qualitative benefits that come with consistency and streamlining work.

  Your products and services would be deployed through an optimized path with security and governance built-in, rather than a sluggish, manual and error-prone process. After all, isn’t that the dream of DevOps, GitOps, and Cloud Native?

  Introducing IaC

  Let’s use a different analogy. Think of IaC as the blueprint for resources (such as servers, databases, networking components, or PaaS services) that host our software and services.

  If you were architecting a hospital or a school, you wouldn’t use the same overall blueprint for both scenarios as they serve entirely different purposes with significantly different requirements. But there are likely building blocks or foundations that can be reused across the two designs.

  IaC solutions, such as HCL, allow us to define and reuse these building blocks, similarly to how we reuse methods, modules, and package libraries in software development. With it being IaC, we can start adopting the same recommended practices for infrastructure that we use when collaborating and deploying on applications.

  After all, we know that teams that adopt DevOps methodologies will see improved productivity, cloud-enabled scalability, collaboration, and security.

  A better way to deliver

  With that context, let’s explore the tangible benefits that we gain in codifying our infrastructure and how they can help us transform our traditional Ops culture.

  Storing code in repositories

  Let’s start with the lowest-hanging fruit. With it being IaC, we can start storing infrastructure and architectural patterns in source code repositories such as GitHub. This gives us a single source of truth with a complete version history. This allows us to easily rollback changes if needed, or deploy a specific version of the truth from history.

  Teams across the enterprise can collaborate in separate branches in a Git repository. Branches allow teams and individuals to be productive in “their own space” and not have to worry about negatively impacting the in-progress work of other teams, away from the “production” source of truth (typically, the main branch).

  Terraform modules, the reusable building blocks mentioned in the last section, are also stored and versioned in Git repositories. From there, modules can be imported to the private registry in Terraform Cloud to make them easily discoverable by all teams. When a new release version is tagged in GitHub, it is automatically updated in the registry.

  Collaborate early and often

  As we discussed above, teams can make changes in separate branches to not impact the current state. But what happens when you want to bring those changes to the production codebase? If you’re unfamiliar with Git, then you may not have heard of a pull request before. As the name implies, we can “pull” changes from one branch into another.

  Pull requests in GitHub are a great way to collaborate with other users in the team, being able to get peer reviews so feedback can be incorporated into your work. The pull request process is deliberately very social, to foster collaboration across the team.

  In GitHub, you could consider setting branch protection rules so that direct changes to your main branch are not allowed. That way, all users must go through a pull request to get their code into production. You can even specify the minimum number of reviewers needed in branch protection rules.

  Tip: you could use a special type of file, the CODEOWNERS file in GitHub, to automatically add reviewers to a pull request based on the files being edited. For example, all HCL files may need a review by the core infrastructure team. Or IaC configurations for line of business core banking systems might require review by a compliance team.

  Unlike Change Advisory Boards, which typically take place on a specified cadence, pull requests become a natural part of the process to bring code into production. The quality of the decisions and discussions also evolves. Rather than being a “yes/no” decision with recommendations in an external system, the context and recommendations can be viewed directly in the pull request.

  Collaboration is also critical in the provisioning process, and GitHub’s integrations with Terraform Cloud will help you scale these processes across multiple teams. Terraform Cloud offers workflow features like secure storage for your Terraform state and direct integration with your GitHub repositories for a turnkey experience around the pull request and merge lifecycle.

  Bringing automated quality reviews into the process

  Building on from the previous section, pull requests also allow us to automatically check the quality of the changes that are being proposed. It is common in software to check that the application still compiles correctly, that unit tests pass, that no security vulnerabilities are introduced, and more.

  From an IaC perspective, we can bring similar automated checks into our process. This is achieved by using GitHub status checks and gives us a clear understanding of whether certain criteria has been met or not.

  GitHub Actions are commonly used to execute some of these automated checks in pull requests on GitHub. To determine the quality of IaC, you could include checks such as:

  • Validating that the code is syntactically correct (for example, Terraform validate).
  • Linting the code to ensure a certain set of standards are being followed (for example, TFLint or Terraform format).
  • Static code analysis to identify any misconfigurations in your infrastructure at “design time” (for example, tfsec or terrascan).
  • Relevant unit or integration tests (using tools such as Terratest).
  • Deploying the infrastructure into a “smoke test”environment to verify that the infrastructure configuration (along with a known set of parameters) results deploy into a desired state.

  Getting started with Terraform on GitHub is easy. Versions of Terraform are installed on our Linux-based GitHub-hosted runners, and HashiCorp has an official GitHub Action to set up Terraform on a runner using a Terraform version that you specify.

  Compliance as an automated check

  We recently blogged about building compliance, security, and audit into your delivery pipelines and the benefits of this approach. When you add IaC to your existing development pipelines and workflows, you’ll have the ability to describe previously manual compliance testing and artifacts as code directly into your HCL configurations files.

  A natural extension to IaC, policy as code allows your security and compliance teams to centralize the definitions of your organization’s requirements. Terraform Cloud’s built-in support for the HashiCorp Sentinel and Open Policy Agent (OPA) frameworks allows policy sets to be automatically ingested from GitHub repositories and applied consistently across all provisioning runs. This ensures policies are applied before misconfigurations have a chance to make it to production.

  An added bonus mentioned in another recent blog is the ability to leverage AI-powered compliance solutions to optimize your delivery even more. Imagine a future where generative AI could create compliance-focused unit-tests across your entire development and infrastructure delivery pipeline with no manual effort.

  Security in the background

  You may have heard of Dependabot, our handy tool to help you keep your dependencies up to date. But did you know that Dependabot supports Terraform? That means you could rely on Dependabot to help keep your Terraform provider and module versions up to date.

  Checks complete, time to deploy

  With the checks complete, it’s now time for us to deploy our new infrastructure configuration! Branching and deployment strategies is beyond the scope of this post, so we’ll leave that for another discussion.

  However, GitHub Actions can help us with the deployment aspect as well! As we explained earlier, getting started with Terraform on GitHub is easy. Versions of Terraform are installed on our Linux-based GitHub-hosted runners, and HashiCorp has an official GitHub Action to set up Terraform on a runner using a Terraform version that you specify.

  But you can take this even further! In Terraform, it is very common to use the command terraform plan to understand the impact of changes before you push them to production. terraform apply is then used to execute the changes.

  Reviewing environment changes in a pull request

  HashiCorp provides an example of automating Terraform with GitHub Actions. This example orchestrates a release through Terraform Cloud by using GitHub Actions. The example takes the output of the terraform plan command and copies the output into your pull request for approval (again, this depends on the development flow that you’ve chosen).

  Reviewing environment changes using GitHub Actions environments

  Let’s consider another example, based on the example from HashiCorp. GitHub Actions has a built-in concept of environments. Think of these environments as a logical mapping to a target deployment location. You can associate a protection rule with an environment so that an approval is given before deploying.

  So, with that context, let’s create a GitHub Action workflow that has two environments—one which is used for planning purposes, and another which is used for deployment:

  name: 'Review and Deploy to EnvironmentA'
  on: [push]
  
  jobs:
    review:
      name: 'Terraform Plan'
      environment: environment_a_plan
      runs-on: ubuntu-latest
  
      steps:
        - name: 'Checkout'
          uses: actions/checkout@v2
  
        - name: 'Terraform Setup'
          uses: hashicorp/setup-terraform@v2
          with:
            cli_config_credentials_token: ${{ secrets.TF_API_TOKEN }}
  
        - name: 'Terraform Init'
          run: terraform init
  
  
        - name: 'Terraform Format'
          run: terraform fmt -check
  
        - name: 'Terraform Plan'
          run: terraform plan -input=false
  
    deploy:
      name: 'Terraform'
      environment: environment_a_deploy
      runs-on: ubuntu-latest
      needs: [review]
  
      steps:
        - name: 'Checkout'
          uses: actions/checkout@v2
  
        - name: 'Terraform Setup'
          uses: hashicorp/setup-terraform@v2
          with:
            cli_config_credentials_token: ${{ secrets.TF_API_TOKEN }}
  
        - name: 'Terraform Init'
          run: terraform init
  
        - name: 'Terraform Plan'
          run: terraform apply -auto-approve -input=false
  

  Before executing the workflow, we can create an environment in the GitHub repository and associate protection rules with the environment_a_deploy. This means that a review is required before a production deployment.

  Learn more

  Check out HashiCorp’s Practitioner’s Guide to Using HashiCorp Terraform Cloud with GitHub for some common recommendations on getting started. And find out how we at GitHub are using Terraform to deliver mission-critical functionality faster and at lower cost.

  
  
  https://github.blog/2023-02-28-how-to-build-a-consistent-workflow-for-development-and-operations-teams/
• 

  10 things you didn’t know you could do with GitHub Codespaces
  • sorangutan  

  1
  1
  Posts
  12
  Views

  

  Ever feel like you’re coding on a plane mid-flight? When I first learned to code about five years ago, my laptop was painstakingly slow, but I couldn’t afford a better one. That’s why I relied on browser-based IDEs like jsbin.com to run my code.

  Now fast forward to today, where GitHub Codespaces provides a fully-fledged, browser-based Integrated Development Environment (IDE) on a virtual machine. This means you can code without draining your local machine’s resources. Cloud-powered development is a game-changer for folks with less powerful machines, but that barely scratches the surface of GitHub Codespaces’ versatility.

  In this blog post, we’ll discuss a few ways that you can get the most out of GitHub Codespaces!

  Generate AI images

  You can run Stable Diffusion with GitHub Codespaces. Like DALL-E and Midjourney, Stable Diffusion is one of many machine-learning models using deep learning to convert text into art.

  Stable Diffusion takes the following steps to convert text into art:

  • AI receives an image
  • AI adds noise to the image until the image becomes completely unrecognizable. (Noise is another way of saying pixelated dots.)
  • AI removes the noise until it produces a clear, high-quality image
  • AI learns from a database called LAION-Aesthetics. The database has image-text pairs to learn to convert text into images.

  This entire process is resource-intensive! Experts recommend using a computer with a powerful graphics processing unit (GPU) to run data-heavy tasks like Stable Diffusion. However, not everyone has a computer with that type of computing power (including me), so instead, I use GitHub Codespaces.

  Since your codespace is hosted on a virtual machine, you can set the machine type from 2-core to 32-core. You can request access to a GPU-powered codespace if you need a more powerful machine. This means a machine learning engineer can use an iPad or Chromebook to perform data-heavy deep learning computations via GitHub Codespaces.

  Check out my DEV post and repository to learn more about how I generated art inside my codespace.

  Below you can see the code and AI image that GitHub Codespaces helped me produce:

  
  from torch import autocast
  
  # Change prompt for image here!
  prompt = "a cartoon black girl with cotton candy hair and a pink dress standing in front of a pink sky with cotton candy clouds"
  with autocast(device):
    image = pipe(prompt, height=768, width=768).images[0]
  image
  

  

  Manage GitHub Codespaces from the command line

  Back in the 2000s, I would inadvertently close my documents, or my computer would suddenly shut down, and my work would be lost forever. Fortunately, it’s 2023, and many editors protect against that, including GitHub Codespaces. If you close a codespace, it will save your work even if you forget to commit your changes.

  However, if you’re done with a codespace because you pushed your code to a repository, you can feel free to delete it. GitHub Codespace management is important for the following reasons:

  • You don’t want to get confused with all your GitHub Codespaces and accidentally delete the wrong one.
  • By default, inactive GitHub Codespaces automatically delete every 30 days.

  You can manage your GitHub Codespaces through the GitHub UI or GitHub CLI. The GitHub CLI allows you to do things like:

  Create a codespace

  gh codespace create -r OWNER/REPO_NAME [-b BRANCH]
  

  List all your codespaces

  gh codespace list
  

  Delete a codespace:

  gh codespace delete -c CODESPACE-NAME
  

  Rename a codespace

  gh codespace edit -c CODESPACE-NAME -d DISPLAY-NAME
  

  Change the machine type of a codepace

  gh codespace edit -m MACHINE-TYPE-NAME
  

  This is not an exhaustive list of all ways you can manage GitHub Codespaces via the CLI. You learn more about managing your codespaces from the command line here!

  Pair program with a teammate

  Pair programming when you’re working remotely is challenging. It’s harder to share your screen and your code when you’re not sitting next to your teammate. However, it’s worthwhile when it works because it helps both parties improve their communication and technical skills. Installing the Live Share extension and forward ports can make remote pair programming easier with GitHub Codespaces.

  Share your forwarded ports

  You can share the URL of your locally hosted web app and make it available to your teammates or collaborators. If you right-click on the port you’d like to share, you’ll see the option to switch the port visibility from “Private” to “Public.” If you copy the value in the local address and share it with necessary collaborators, they can view, test, and debug your locally hosted web app even though they’re in a different room than you.

  

  Live Share

  The Live Share extension allows you and your teammate(s) to type in the same project simultaneously. It highlights your teammate’s cursor versus your cursor, so you can easily identify who is typing, and it’s completely secure!

  

  Pair program with AI

  Visual Studio Code’s marketplace offers a wide variety of extensions compatible with GitHub Codespaces, including GitHub Copilot. If you need to exchange ideas while you code, but there’s no one around, try installing the GitHub Copilot extension to pair program with AI. GitHub Copilot is an AI-powered code completion tool that helps you write code faster by suggesting code snippets and completing code for you.

  Beyond productivity through code completion, GitHub Copilot empowers developers of varied backgrounds. One of my favorite GitHub Copilot tools is “Hey, GitHub,” a hands-free, voice-activated AI programmer. Check out the video below to learn how “Hey, GitHub” improves the developer experience for folks with limited physical dexterity or visual impairments.

  ---

  ---

  Learn about more use cases for GitHub Copilot here.

  

  Teach people to code

  While educating people about coding via bootcamps, classrooms, meetups, and conferences, I’ve learned that teaching people how to code is hard. Sometimes in workshops, attendees spend most of the time trying to set up their environment so that they can follow along. However, teaching people to code or running a workshop in GitHub Codespaces creates a better experience for all participants. Instead of expecting beginner developers to understand how to clone repositories to work with a template, they can open up a codespace and work in a development environment you’ve configured. Now, you can have the peace of mind that everyone has the same environment as you and can easily follow along. This reduces tons of time, stress, and chaos.

  We made GitHub Codespaces more accessible for teachers and students by offering 180 free hours of usage (equivalent to five assignments per month for a class of 50). Check out my tutorial on configuring GitHub Codespaces and leveraging extensions like CodeTour to run self-guided workshops.

  Learn how CS50, Harvard’s largest class, uses GitHub Codespaces to teach students to code.

  Learn a new framework

  When you’re learning to code, it’s easy to over-invest your time into following tutorials. Learning is often more effective when you balance tutorial consumption with building projects. GitHub Codespaces’ quickstart templates are a speedy and efficient way to learn a framework.

  Quickstart templates include boilerplate code, forwarded ports, and a configured development container for some of the most common application frameworks, including Next.js, React.js, Django, Express, Ruby on Rails, Preact, Flask, and Jupyter Notebook. Templates provide developers with a sandbox to build, test, and debug applications in a codespace. It only takes one click to open a template and experiment with a new framework.

  Experimenting with a framework in a codespace can help developers better understand a framework’s structure and functionalities, leading to better retention and mastery. For example, I’m not familiar with Jupyter Notebook, but I’ve used a Jupyter Notebook template in GitHub Codespaces. The boilerplate code helped me to experiment with and better understand the structure of a Jupyter Notebook. I even built a small project with it!

  Check out this blog post and template, where we use GitHub Codespaces to guide developers through writing their first lines of React.

  

  Store environment variables

  In the past, I’ve had multiple moments where I’ve accidentally shared or mishandled my environment variables. Here are a few cringe-worthy moments where I’ve mishandled environment variables:

  • I’ve shown them to an audience during a live demo.
  • I’ve pushed them to GitHub.
  • I couldn’t figure out a great way to share the values with teammates.

  To avoid these moments, I could’ve used GitHub Codespaces secrets. You can store API Keys, environment variables, and secret credentials in your GitHub Codespaces secrets for each of your GitHub Codespaces.

  After you store them in your GitHub Codespaces secrets, you can refer to the environment variables in your code as: process.env.{SUPER_SECRET_API_KEY}.

  Check out this tutorial for more information on securely storing your environment variables with GitHub Codespaces

  

  Onboard developers faster

  Typically, software engineers are responsible for setting up their local environment when they join a team. Local environment setup can take a day or a week, depending on the quality of the documentation. Fortunately, organizations can automate the onboarding process using GitHub Codespaces to configure a custom environment. When a new engineer joins the team, they can open a codespace and skip the local environment setup because the required extensions, dependencies, and environment variables exist within the codespace.

  In 2021, the GitHub Engineering Team migrated from our macOS model to GitHub Codespaces for the development of GitHub.com. After over 14 years, our repository accrued almost 13 GB on disk. It took approximately 45 minutes to clone, set up dependencies, and bootstrap our repository. After fully migrating to GitHub Codespaces, it only takes 10 seconds to launch GitHub.com in a preconfigured, reliable codespace.

  However, you don’t have to take our word for it. Ketan Shah, Manager of Information Services at TELUS, a telecommunications company, attests, “GitHub shaves off entire days from the new employee onboarding process. New developers are up and running within minutes because everything is all in one place.”

  Learn more about GitHub Codespaces can quickly onboard developers:

  • How TELUS uses GitHub Codespaces
  • How Elanco uses GitHub Codespaces
  • How your team can use GitHub Codespace to configure a positive onboarding experience with development containers

  Conduct technical interviews

  Take-home projects, technical screens, and live coding are often painful, but necessary parts of the interview experience for software developers, but using GitHub Codespaces can help ease some of the difficulties. By providing candidates with a well-configured and reliable environment, interviewers can help reduce anxiety and improve performance. With GitHub Codespaces, candidates no longer have to worry about setting up their environment. Interviewers can use GitHub codespaces to provide real-time feedback and collaborate using built-in tools like Visual Studio Code Live Share. Furthermore, GitHub Codespaces offers a secure and isolated environment for the interview process, ensuring that sensitive information and data remain protected.

  Learn how various engineering teams at GitHub use GitHub Codespaces to conduct interviews.

  Code in the cloud with your preferred editor

  I love Visual Studio Code; it’s my primary editor. However, depending on your role and other factors, you might have a different preference. GitHub Codespaces supports the following editors, in addition to Visual Studio Code:

  • Jupyter Notebook
  • IntelliJ IDEA
  • RubyMine
  • GoLand
  • PyCharm
  • PhpStorm
  • WebStorm

  But, wait, there’s more!

  GitHub Codespaces’ superpower is that you can code from any device and get a standardized environment as long as you have internet. However, it also includes features that make software development workflows easier. With 60 hours of free access per month for GitHub users and 90 hours per month for GitHub Pro users, there’s no reason not to try it out and experience its benefits for yourself. So why not give it a try today?

  
  
  https://github.blog/2023-02-28-10-things-you-didnt-know-you-could-do-with-github-codespaces/
• 

  3 ways to meet compliance needs without slowing down agility
  • sorangutan  

  1
  1
  Posts
  14
  Views

  

  In the previous blog, Setting the foundations for compliance, we set the groundwork for developer-enabled compliance that will keep your teams happy, in the flow, secure, compliant, and auditable.

  Today, we’ll walk through three practical ways that you can start meeting your compliance requirements without having to revolutionize or transform the culture in your company—all while enabling your developers to be more productive and happy.

  1. Do the basics well and often

  The first way to start meeting your compliance requirements is often overlooked because it’s so simple. No matter what industry you are in, whether it’s finance, automotive, government, or tech, there are some basic quick compliance wins that may be part of your existing developer workflows and culture.

  Code review

  A key part of writing clean code is code review. But having a repeatable and traceable code review process is also a foundational component of your compliance program. This ensures risks within software delivery are found and mitigated early—and are also auditable for future reviews.

  GitHub makes code review easy, since pull requests are part of the existing workflow that millions of developers use every day. GitHub also makes it easy for compliance testers and auditors to review pull requests with access to immutable audit logs in a central location.

  Separation of duties

  In some enterprises, separation of duties has been defined as a person having too much manual control over transactions or processes. This can lead to apprehension when modern cloud native practices are introduced.

  Thankfully, there is guidance from the Industry that supports a more modern approach to separation of duties. The PCI-DSS requirements guide avoids the term person and provides a more cloud native friendly approach by focusing on functions and accounts:

  “The purpose of this requirement is to separate the development and test functions from the production functions. For example, a developer can use an administrator-level account with elevated privileges in the development environment and have a separate account with user-level access to the production environment.”

  This approach aligns well with the 12 factor application methodology for cloud native. The Build, release, run factor explanation states, “Each release must enforce a strict separation across the build, release, and run stages. Each should be tagged with a unique ID and support the ability to roll back.” Teams can fully automate their delivery to production, without having to worry about a person manually exceeding too much control, as long as there is separation of functions and traceability back to unique IDs. With the added assurance that they are aligned to industry best practices and requirements, such as PCI-DSS.

  To enable separation of duties, you have to have a clear identity and access management strategy. Thankfully, we don’t have to reinvent the wheel. GitHub Enterprise has several options to help you manage access to your overall development environment:

  • You can configure SAML single sign-on for your enterprise. This is an additional check, allowing you to confirm the authenticity of your users against your own identity provider (while still using your own GitHub account).
  • You can then synchronize team memberships with groups in your identity provider. As a result, group membership changes in your identity provider update the team membership (and therefore associated access) in GitHub.
  • Alternatively, you could adopt GitHub Enterprise Managed Users (EMUs). This is a flavor of GitHub Enterprise where you can only log in with an account that is centrally managed through your identity provider. The user does not have to log in to GitHub with a personal account and use single sign on to access company resources. (For more information on this, check out this blog post on exploring EMUs and the benefits they can bring.)

  AI-powered compliance

  In our last blog we briefly covered AI-enabled compliance and some of the existing opportunities for security, manufacturing, and banks. There are also several other opportunities on the horizon that could further optimize the basics of compliance.

  It is entirely possible that a generative AI tool could soon be leveraged to help ensure that separation of duties is enforced in a declarative DevOps workflow by creating unit tests on its own. Because of the non-deterministic nature of generative AI, each time it runs it may have a different result and the unit tests may include risk scenarios that nobody has thought of yet. This can add an amazing level of true risk-based compliance.

  One of the major benefits of addressing compliance often in your delivery is an increased level of trust. But quantifying trust can be extremely difficult—especially within a regulated industry that wants to leverage deep learning solutions. There is work being driven by AI results to help provide trust quantification for these types of solutions, which will not only enable continuous compliance but will also help enterprises implement new technologies that can increase business value.

  Dependency management

  Companies are increasing their reliance on open source software in their supply chain. As a result, optimized, repeatable, and audible controls for dependency management are becoming a cornerstone of compliance programs across industries.

  From a GitHub perspective, Dependabot can provide confidence that your supply chain is secure. When Dependabot is configured, developers are alerted as soon as a security threat is found and gives them the ability to take action in their normal workflows.

  By leveraging Dependabot, you will receive an immutable and auditable alert when a repository uses a software dependency with a known vulnerability. Pull requests can be triggered by either your developers or security teams, which give you another set of auditable artifacts for future review.

  Approvals

  While most organizations have approval processes, they can be slow and occur too late in the process. Google explains that peer reviews can help to streamline the change approval process, based on results from the DORA 2019 state of DevOps report.

  There could be many control points in your delivery pipeline that may require approval, such as sign-off of requirements, testing results, security reviews, or production operational readiness confirmations before a release.

  Depending on your internal structure and alignment, you may require teams to provide sign-off at different stages throughout the process. In your build and test stage, you can use manual approvals in a pull request to gather the needed reviews.

  Once your builds and tests are complete, it’s time to release your code into your infrastructure. Talking about deployment patterns and strategies is beyond the scope of this post. However, you may require approvals as part of the deployment process.

  To obtain these approvals, you should use environments for your deployments. Environments are used to describe a deployment target (such as staging, testing, and production). By using environments, you can require a manual approval to take place before GitHub Actions begins the deployment job.

  In both instances, remember that there is a tradeoff when deciding the number of approvals required. Setting the number of required reviews too high means that you may impact your pace of delivery, as manual intervention is required. Where possible, consider automating checks to reduce the manual overhead, thereby optimizing your overall agility.

  2. Have a common understanding of concepts and terms

  Again, this may be overlooked since it sounds so obvious. But there are probably many concepts and terms that developers, DevOps and cloud native practitioners use on a daily basis that may be totally incomprehensible to compliance testers and auditors.

  Back in 2015, the author of The Phoenix Project, Gene Kim, and several other authors, created the DevOps Audit Defense Toolkit. As we mentioned in our first blog, the goal of this document was to “educate IT management and practitioners on the audit process so they can demonstrate to auditors they understand the business risks and are properly mitigating those risks.” But where do you start?

  Below is a basic cheat-sheet of terms for GitHub related control objectives and a mapping to the world of compliance, regulations, risk, and audit. This could be a good place to start building a common understanding between developers and your compliance and audit friends.

  Objective	Control	Financial Reporting	Industry Frameworks
  The Code Review Control ensures that security requirements have been addressed and that the code is understandable, maintainable, and properly formatted.	Pull requests let you tell others about changes you’ve pushed to a branch in a repository on GitHub.Once a pull request is opened, you can discuss and review the potential changes with collaborators and add follow-up commits before your changes are merged into the base branch.	SOX: Change Management COSO: Control Activities—Conduct application change management.	NIST Cyber: DE.CM-4 —Malicious code is detected. PCI-DSS: 6.3.2: Review all custom code for vulnerabilities, manually or via automation. SLSA: Two-person review is an industry best practice for catching mistakes and deterring bad behavior.
  Controls and processes should be in place to scan code repositories for passwords and security tokens. Prevention measures should be enforced to ensure secrets do not reach production.	Secret scanning will scan your entire Git history on all branches present in your GitHub repository for secrets. GitHub push protection will check for high-confidence secrets as developers push code and block the push if a secret is identified.	SOX: IT Security COSO: Control Activities—Improve security	NIST Cyber: PR.DS-5: Protections against data leaks are implemented. PCI-DSS: 6.5.3: Protect against all insufficiently secure cryptographic key storage.

  To help bring this together, there is a great example of the benefits of a common understanding between developers and compliance testers and auditors in the latest Forrester report on the economic impact of GitHub Enterprise Cloud and Advanced Security. The report highlights one of the benefits of an automated documentation process that both helps developers and auditors:

  “These new, standardized documentation structures made it easier for auditors to find and compile the documentation necessary to be compliant. This helped the interviewees’ organizations save time preparing for industry compliance and security audits.”

  3. Helping developers understand where they fit into the three lines of defense

  The three lines of defense is a model used for risk management and governance. It helps clarify the roles and responsibilities of teams involved in addressing risk.

  Think of the engineering team and engineering leads as the first line of defense, as they continue shipping software to their end-users. Making sure this team has the appropriate level of skills to identify risk within their solution is crucial.

  The second line of defense is typically achieved at scale through frameworks, policies, and tools at an organizational level. This acts as oversight for the first line of defense on how well they are implementing risk mitigation techniques and consistently measuring and defining risk across the organization.

  Internal audit is the third line of defense. Just like a red team and blue team compliment each other, you can think of internal audit as the final puzzle piece in the three lines of defense. They evaluate the effectiveness of risk management and governance, working with senior management and external regulators to raise awareness of the controls being executed.

  Let’s use an analogy. A hockey team is not just structured to score goals but also prevent goals being scored against them.

  • Forwards: they are there to score and assist on goals. But they can be called on to work with the other positions in defense. Think of them as the first line of defense. Their job is to create solutions which provide business value, but are also secure.
  • Defenders: their main role is clearly preventing goals. But they partner with the forwards on the current priority (offense or defense). They are like the second line of defense, providing risk oversight and collaborating with the engineering teams.
  • Goaltender: the last line of defense. They can be thought of as the equivalent of an internal audit. They are independent of the forwards and defenders with different responsibilities, tools, and perspective.

  Hockey teams that have very strong forwards but weak defenders and goalkeepers are rarely successful. Teams with a strong defense but weak offense are rarely successful, either. It takes all three aspects working in harmony to be successful.

  This applies in business, too. If your solutions meet your customers’ requirements but are insecure, your business will fail. If your solutions are secure but aren’t user friendly or providing value, it will result in failure. Success is achieved by finding the right balance of value and risk management.

  Developers can see that they are there to score goals for their team; creating the software that runs our world. But they need to support the defensive capabilities of the team, ensuring their software is secure. Their success is dependent on the success of the wider team, without having to take on additional responsibilities.

  Next steps

  We’ve taken a whistle-stop tour on how you can bring compliance into your development flow. From branch protection rules and pull requests to CODEOWNERS and environment approvals, there are several GitHub capabilities that can help you naturally focus on compliance.

  This is only one step in solving the problem. A common language between compliance and DevOps practitioners is crucial in demonstrating the implemented measures. With that common language, it is clear that everyone must think about compliance; engineering teams are a part of the three lines of defense.

  Next up in the series, we’ll be talking about how to ensure compliance in developer workflows.

  Ready to increase developer velocity and collaboration while remaining secure and compliant? See how GitHub Enterprise can help.

  
  
  https://github.blog/2023-02-24-3-ways-to-meet-compliance-needs-without-slowing-down-agility/
• 

  The technology behind GitHub’s new code search
  • sorangutan  

  1
  1
  Posts
  16
  Views

  

  From launching our technology preview of the new and improved code search experience a year ago, to the public beta we released at GitHub Universe last November, there’s been a flurry of innovation and dramatic changes to some of the core GitHub product experiences around how we, as developers, find, read, and navigate code.

  One question we hear about the new code search experience is, “How does it work?” And to complement a talk I gave at GitHub Universe, this post gives a high-level answer to that question and provides a small window into the system architecture and technical underpinnings of the product.

  So, how does it work? The short answer is that we built our own search engine from scratch, in Rust, specifically for the domain of code search. We call this search engine Blackbird, but before I explain how it works, I think it helps to understand our motivation a little bit. At first glance, building a search engine from scratch seems like a questionable decision. Why would you do that? Aren’t there plenty of existing, open source solutions out there already? Why build something new?

  To be fair, we’ve tried and have been trying, for almost the entire history of GitHub, to use existing solutions for this problem. You can read a bit more about our journey in Pavel Avgustinov’s post, A brief history of code search at GitHub, but one thing sticks out: we haven’t had a lot of luck using general text search products to power code search. The user experience is poor, indexing is slow, and it’s expensive to host. There are some newer, code-specific open source projects out there, but they definitely don’t work at GitHub’s scale. So, knowing all that, we were motivated to create our own solution by three things:

  1. We’ve got a vision for an entirely new user experience that’s about being able to ask questions of code and get answers through iteratively searching, browsing, navigating, and reading code.
  2. We understand that code search is uniquely different from general text search. Code is already designed to be understood by machines and we should be able to take advantage of that structure and relevance. Searching code also has unique requirements: we want to search for punctuation (for example, a period or open parenthesis); we don’t want stemming; we don’t want stop words to be stripped from queries; and, we want to search with regular expressions.
  3. GitHub’s scale is truly a unique challenge. When we first deployed Elasticsearch, it took months to index all of the code on GitHub (about 8 million repositories at the time). Today, that number is north of 200 million, and that code isn’t static: it’s constantly changing and that’s quite challenging for search engines to handle. For the beta, you can currently search almost 45 million repositories, representing 115 TB of code and 15.5 billion documents.

  At the end of the day, nothing off the shelf met our needs, so we built something from scratch.

  Just use grep?

  First though, let’s explore the brute force approach to the problem. We get this question a lot: “Why don’t you just use grep?” To answer that, let’s do a little napkin math using ripgrep on that 115 TB of content. On a machine with an eight core Intel CPU, ripgrep can run an exhaustive regular expression query on a 13 GB file cached in memory in 2.769 seconds, or about 0.6 GB/sec/core.

  We can see pretty quickly that this really isn’t going to work for the larger amount of data we have. Code search runs on 64 core, 32 machine clusters. Even if we managed to put 115 TB of code in memory and assume we can perfectly parallelize the work, we’re going to saturate 2,048 CPU cores for 96 seconds to serve a single query! Only that one query can run. Everybody else has to get in line. This comes out to a whopping 0.01 queries per second (QPS) and good luck doubling your QPS—that’s going to be a fun conversation with leadership about your infrastructure bill.

  There’s just no cost-effective way to scale this approach to all of GitHub’s code and all of GitHub’s users. Even if we threw a ton of money at the problem, it still wouldn’t meet our user experience goals.

  You can see where this is going: we need to build an index.

  A search index primer

  We can only make queries fast if we pre-compute a bunch of information in the form of indices, which you can think of as maps from keys to sorted lists of document IDs (called “posting lists”) where that key appears. As an example, here’s a small index for programming languages. We scan each document to detect what programming language it’s written in, assign a document ID, and then create an inverted index where language is the key and the value is a posting list of document IDs.

  Forward index

  Doc ID	Content
  1	def lim puts “mit” end
  2	fn limits() {
  3	function mits() {

  Inverted index

  Language	Doc IDs (postings)
  JavaScript	3, 8, 12, …
  Ruby	1, 10, 13, …
  Rust	2, 5, 11, …

  For code search, we need a special type of inverted index called an ngram index, which is useful for looking up substrings of content. An ngram is a sequence of characters of length n. For example, if we choose n=3 (trigrams), the ngrams that make up the content “limits” are lim, imi, mit, its. With our documents above, the index for those trigrams would look like this:

  ngram	Doc IDs (postings)
  lim	1, 2, …
  imi	2, …
  mit	1, 2, 3, …
  its	2, 3, …

  To perform a search, we intersect the results of multiple lookups to give us the list of documents where the string appears. With a trigram index you need four lookups: lim, imi, mit, and its in order to fulfill the query for limits.

  Unlike a hashmap though, these indices are too big to fit in memory, so instead, we build iterators for each index we need to access. These lazily return sorted document IDs (the IDs are assigned based on the ranking of each document) and we intersect and union the iterators (as demanded by the specific query) and only read far enough to fetch the requested number of results. That way we never have to keep entire posting lists in memory.

  Indexing 45 million repositories

  The next problem we have is how to build this index in a reasonable amount of time (remember, this took months in our first iteration). As is often the case, the trick here is to identify some insight into the specific data we’re working with to guide our approach. In our case it’s two things: Git’s use of content addressable hashing and the fact that there’s actually quite a lot of duplicate content on GitHub. Those two insights lead us the the following decisions:

  1. Shard by Git blob object ID which gives us a nice way of evenly distributing documents between the shards while avoiding any duplication. There won’t be any hot servers due to special repositories and we can easily scale the number of shards as necessary.
  2. Model the index as a tree and use delta encoding to reduce the amount of crawling and to optimize the metadata in our index. For us, metadata are things like the list of locations where a document appears (which path, branch, and repository) and information about those objects (repository name, owner, visibility, etc.). This data can be quite large for popular content.

  We also designed the system so that query results are consistent on a commit-level basis. If you search a repository while your teammate is pushing code, your results shouldn’t include documents from the new commit until it has been fully processed by the system. In fact, while you’re getting back results from a repository-scoped query, someone else could be paging through global results and looking at a different, prior, but still consistent state of the index. This is tricky to do with other search engines. Blackbird provides this level of query consistency as a core part of its design.

  Let’s build an index

  Armed with those insights, let’s turn our attention to building an index with Blackbird. This diagram represents a high level overview of the ingest and indexing side of the system.

  

  Kafka provides events that tell us to go index something. There are a bunch of crawlers that interact with Git and a service for extracting symbols from code, and then we use Kafka, again, to allow each shard to consume documents for indexing at its own pace.

  Though the system generally just responds to events like git push to crawl changed content, we have some work to do to ingest all the repositories for the first time. One key property of the system is that we optimize the order in which we do this initial ingest to make the most of our delta encoding. We do this with a novel probabilistic data structure representing repository similarity and by driving ingest order from a level order traversal of a minimum spanning tree of a graph of repository similarity¹.

  Using our optimized ingest order, each repository is then crawled by diffing it against its parent in the delta tree we’ve constructed. This means we only need to crawl the blobs unique to that repository (not the entire repository). Crawling involves fetching blob content from Git, analyzing it to extract symbols, and creating documents that will be the input to indexing.

  These documents are then published to another Kafka topic. This is where we partition² the data between shards. Each shard consumes a single Kafka partition in the topic. Indexing is decoupled from crawling through the use of Kafka and the ordering of the messages in Kafka is how we gain query consistency.

  The indexer shards then take these documents and build their indices: tokenizing to construct ngram indices³ (for content, symbols, and paths) and other useful indices (languages, owners, repositories, etc) before serializing and flushing to disk when enough work has accumulated.

  Finally, the shards run compaction to collapse up smaller indices into larger ones that are more efficient to query and easier to move around (for example, to a read replica or for backups). Compaction also k-merges the posting lists by score so relevant documents have lower IDs and will be returned first by the lazy iterators. During the initial ingest, we delay compaction and do one big run at the end, but then as the index keeps up with incremental changes, we run compaction on a shorter interval as this is where we handle things like document deletions.

  Life of a query

  Now that we have an index, it’s interesting to trace a query through the system. The query we’re going to follow is a regular expression qualified to the Rails organization looking for code written in the Ruby programming language: /arguments?/ org:rails lang:Ruby. The high level architecture of the query path looks a little bit like this:

  

  In between GitHub.com and the shards is a service that coordinates taking user queries and fanning out requests to each host in the search cluster. We use Redis to manage quotas and cache some access control data.

  The front end accepts the user query and passes it along to the Blackbird query service where we parse the query into an abstract syntax tree and then rewrite it, resolving things like languages to their canonical Linguist language ID and tagging on extra clauses for permissions and scopes. In this case, you can see how rewriting ensures that I’ll get results from public repositories or any private repositories that I have access to.

  And(
      Owner("rails"),
      LanguageID(326),
      Regex("arguments?"),
      Or(
          RepoIDs(...),
          PublicRepo(),
      ),
  )
  

  Next, we fan out and send n concurrent requests: one to each shard in the search cluster. Due to our sharding strategy, a query request must be sent to each shard in the cluster.

  On each individual shard, we then do some further conversion of the query in order to lookup information in the indices. Here, you can see that the regex gets translated into a series of substring queries on the ngram indices.

  and(
    owners_iter("rails"),
    languages_iter(326),
    or(
      and(
        content_grams_iter("arg"),
        content_grams_iter("rgu"),
        content_grams_iter("gum"),
        or(
          and(
           content_grams_iter("ume"),
           content_grams_iter("ment")
          )
          content_grams_iter("uments"),
        )
      ),
      or(paths_grams_iter…)
      or(symbols_grams_iter…)
    ), 
    …
  )
  

  If you want to learn more about a method to turn regular expressions into substring queries, see Russ Cox’s article on Regular Expression Matching with a Trigram Index. We use a different algorithm and dynamic gram sizes instead of trigrams (see below³). In this case the engine uses the following grams: arg,rgu, gum, and then either ume and ment, or the 6 gram uments.

  The iterators from each clause are run: and means intersect, or means union. The result is a list of documents. We still have to double check each document (to validate matches and detect ranges for them) before scoring, sorting, and returning the requested number of results.

  Back in the query service, we aggregate the results from all shards, re-sort by score, filter (to double-check permissions), and return the top 100. The GitHub.com front end then still has to do syntax highlighting, term highlighting, pagination, and finally we can render the results to the page.

  Our p99 response times from individual shards are on the order of 100 ms, but total response times are a bit longer due to aggregating responses, checking permissions, and things like syntax highlighting. A query ties up a single CPU core on the index server for that 100 ms, so our 64 core hosts have an upper bound of something like 640 queries per second. Compared to the grep approach (0.01 QPS), that’s screaming fast with plenty of room for simultaneous user queries and future growth.

  In summary

  Now that we’ve seen the full system, let’s revisit the scale of the problem. Our ingest pipeline can publish around 120,000 documents per second, so working through those 15.5 billion documents should take about 36 hours. But delta indexing reduces the number of documents we have to crawl by over 50%, which allows us to re-index the entire corpus in about 18 hours.

  There are some big wins on the size of the index as well. Remember that we started with 115 TB of content that we want to search. Content deduplication and delta indexing brings that down to around 28 TB of unique content. And the index itself clocks in at just 25 TB, which includes not only all the indices (including the ngrams), but also a compressed copy of all unique content. This means our total index size including the content is roughly a quarter the size of the original data!

  If you haven’t signed up already, we’d love for you to join our beta and try out the new code search experience. Let us know what you think! We’re actively adding more repositories and fixing up the rough edges based on feedback from people just like you.

  Notes

  ---

  1. To determine the optimal ingest order, we need a way to tell how similar one repository is to another (similar in terms of their content), so we invented a new probabilistic data structure to do this in the same class of data structures as MinHash and HyperLogLog. This data structure, which we call a geometric filter, allows computing set similarity and the symmetric difference between sets with logarithmic space. In this case, the sets we’re comparing are the contents of each repository as represented by (path, blob_sha) tuples. Armed with that knowledge, we can construct a graph where the vertices are repositories and edges are weighted with this similarity metric. Calculating a minimum spanning tree of this graph (with similarity as cost) and then doing a level order traversal of the tree gives us an ingest order where we can make best use of delta encoding. Really though, this graph is enormous (millions of nodes, trillions of edges), so our MST algorithm computes an approximation that only takes a few minutes to calculate and provides 90% of the delta compression benefits we’re going for. ↩
  2. The index is sharded by Git blob SHA. Sharding means spreading the indexed data out across multiple servers, which we need to do in order to easily scale horizontally for reads (where we are concerned about QPS), for storage (where disk space is the primary concern), and for indexing time (which is constrained by CPU and memory on the individual hosts). ↩
  3. The ngram indices we use are especially interesting. While trigrams are a known sweet spot in the design space (as Russ Cox and others have noted: bigrams aren’t selective enough and quadgrams take up too much space), they cause some problems at our scale.

     For common grams like for trigrams aren’t selective enough. We get way too many false positives and that means slow queries. An example of a false positive is something like finding a document that has each individual trigram, but not next to each other. You can’t tell until you fetch the content for that document and double check at which point you’ve done a lot of work that has to be discarded. We tried a number of strategies to fix this like adding follow masks, which use bitmasks for the character following the trigram (basically halfway to quad grams), but they saturate too quickly to be useful.

     We call the solution “sparse grams,” and it works like this. Assume you have some function that given a bigram gives a weight. As an example, consider the string chester. We give each bigram a weight: 9 for “ch”, 6 for “he”, 3 for “es”, and so on.

     Click to view slideshow.

     Using those weights, we tokenize by selecting intervals where the inner weights are strictly smaller than the weights at the borders. The inclusive characters of that interval make up the ngram and we apply this algorithm recursively until its natural end at trigrams. At query time, we use the exact same algorithm, but keep only the covering ngrams, as the others are redundant. ↩ ↩

  
  
  https://github.blog/2023-02-06-the-technology-behind-githubs-new-code-search/