{"id":21629,"date":"2022-10-24T14:40:18","date_gmt":"2022-10-24T21:40:18","guid":{"rendered":"https:\/\/coderpad.io\/?p=21629"},"modified":"2023-06-05T14:00:12","modified_gmt":"2023-06-05T21:00:12","slug":"sql-functions-and-techniques-every-data-person-should-know","status":"publish","type":"post","link":"https:\/\/coderpad.io\/blog\/development\/sql-functions-and-techniques-every-data-person-should-know\/","title":{"rendered":"SQL Functions and Techniques Every Data Person Should Know"},"content":{"rendered":"\n

Writing SQL queries that don’t utilize database features can cost developers a lot. The cost varies from spending more time writing tedious long queries or having a tech debt later on to debug the performance issue coming from the database server.<\/p>\n\n\n\n

PostgreSQL has features that help you select data efficiently. These features can also help minimize the lines you write SQL queries.<\/p>\n\n\n\n

In a previous tutorial, we discussed why and how to use window functions<\/a> in PostgreSQL to aggregate data. However, this tutorial will show you what’s beyond aggregate functions, especially with the HAVING clause. You will also learn about subqueries and different types of JOINs.<\/p>\n\n\n\n

Afterward, you will get familiar with features in PostgreSQL, like inheritance, full-text search, views, and how to handle geospatial data in PostgreSQL.<\/p>\n\n\n\n

I encourage you to apply everything mentioned in this tutorial. You can use the CoderPad sandbox<\/a> with the PostgreSQL playground to practice the commands learned in this article.<\/p>\n\n\n\n

Aggregate functions (with <\/strong>HAVING<\/strong> clause)<\/strong><\/h2>\n\n\n\n

An aggregate function is a function that groups a number of rows to form one record. Examples of aggregate functions in SQL are MAX<\/code>, MIN<\/code>, COUNT<\/code>, and SUM<\/code>. To filter the output of the aggregated result, you need a HAVING<\/code> clause.<\/p>\n\n\n\n

The HAVING<\/code> clause is similar to the WHERE<\/code> clause but for aggregate functions. To make it more clear, see the examples below!<\/p>\n\n\n\n

An employee table has been created for the example queries in this section. Have a look at the employees<\/code> table first before proceeding:<\/p>\n\n\n

\n
\"\"\/
Employees table<\/figcaption><\/figure>\n<\/div>\n\n\n

In the employees<\/code> table above, you want to count the number of employees in each department.<\/p>\n\n\n\n

You can use a GROUP BY<\/code> clause for aggregation:<\/p>\n\n\n

SELECT<\/span> department_id, count<\/span>(*) AS<\/span> num_employees\nFROM<\/span> employees\nGROUP<\/span> BY<\/span> department_id;<\/code><\/span>Code language:<\/span> SQL (Structured Query Language)<\/span> (<\/span>sql<\/span>)<\/span><\/small><\/pre>\n\n
\n
\"\"\/
Number of employees in each department.<\/figcaption><\/figure>\n<\/div>\n\n\n
The number of employees in each department designated by their ID is shown in the image above.<\/p>\n\n\n\n
What if you want to get the number of employees in the engineering department?<\/p>\n\n\n\n
To answer this question, first, retrieve the id of the engineering department from the department table using this query:<\/p>\n\n\n
SELECT<\/span> *\nFROM<\/span> departments;<\/code><\/span>Code language:<\/span> SQL (Structured Query Language)<\/span> (<\/span>sql<\/span>)<\/span><\/small><\/pre>\n\n
\n
\"\"\/
IDs of departments in the database.<\/figcaption><\/figure>\n<\/div>\n\n\n
From the image above, the engineering department id is 2. If you filter the department_id<\/code> that equals 2 in the first query, you’ll get the number of employees in the engineering department:<\/p>\n\n\n
SELECT<\/span> department_id, count<\/span>(*) AS<\/span> num_employees\nFROM<\/span> employees\nGROUP<\/span> BY<\/span> department_id\nHAVING<\/span> department_id = 2<\/span>;<\/code><\/span>Code language:<\/span> SQL (Structured Query Language)<\/span> (<\/span>sql<\/span>)<\/span><\/small><\/pre>\n\n
\n
\"\"\/
Number of employees in the engineering department.<\/figcaption><\/figure>\n<\/div>\n\n\n
See another example where you want to get all departments where we have just a single employee. To do that, run this query:<\/p>\n\n\n
SELECT<\/span> department_id, count<\/span>(*) AS<\/span> num_employees\nFROM<\/span> employees\nGROUP<\/span> BY<\/span> department_id\nHAVING<\/span> count<\/span>(*) = 1<\/span>;<\/code><\/span>Code language:<\/span> SQL (Structured Query Language)<\/span> (<\/span>sql<\/span>)<\/span><\/small><\/pre>\n\n
\n
\"\"\/
Departments with a single employee.<\/figcaption><\/figure>\n<\/div>\n\n\n
So the previous two queries showed how to use the HAVING<\/code> clause. This is the clause we use to filter an aggregate function. The HAVING<\/code> clause is followed by an aggregate condition. You can filter the column name by using =<\/code>, <<\/code>, ><\/code>, or any comparison operator<\/a>. You can also use an aggregate function in that condition, like the count()<\/code> function in the previous query.<\/p>\n\n\n\n
Subqueries<\/strong><\/h2>\n\n\n\n
Sometimes your queries must have more than one SELECT<\/code> statement. In such cases, you need subqueries<\/em>.<\/p>\n\n\n\n
Suppose you want to get employees’ salaries that are higher than the average. So you go ahead and get the average salary:<\/p>\n\n\n
SELECT<\/span> AVG<\/span>(salary)\nFROM<\/span> employees;<\/code><\/span>Code language:<\/span> SQL (Structured Query Language)<\/span> (<\/span>sql<\/span>)<\/span><\/small><\/pre>\n\n
\n
\"\"\/
Average salary in the employees table.<\/figcaption><\/figure>\n<\/div>\n\n\n
and then take note of that number and put it in this query:<\/p>\n\n\n
SELECT<\/span> *\nFROM<\/span> employees\nWHERE<\/span> salary > 35000<\/span>;<\/code><\/span>Code language:<\/span> SQL (Structured Query Language)<\/span> (<\/span>sql<\/span>)<\/span><\/small><\/pre>\n\n\n
The problem in this process is that you have an intermediate step. You want to get rid of that manual step and do a one-query. That’s why you need a subquery which is, in this case, an internal query to calculate the average salary so that you can filter by it. Use a subquery as follows:<\/p>\n\n\n
SELECT<\/span> *\nFROM<\/span> employees\nWHERE<\/span> salary > (\n\u00a0 SELECT<\/span> AVG<\/span>(salary)\n\u00a0 FROM<\/span> employees\n);<\/code><\/span>Code language:<\/span> SQL (Structured Query Language)<\/span> (<\/span>sql<\/span>)<\/span><\/small><\/pre>\n\n\n
So now selecting the average has become a subquery that we filtered the salary column with.<\/p>\n\n\n\n
The previous two queries provide the same output:<\/p>\n\n\n
\n
\"\"\/
Employees with salary greater than the average.<\/figcaption><\/figure>\n<\/div>\n\n\n
You can also use subqueries to check the existence of records instead of filtering their values. Suppose you want to get employees who are not currently working on a project. You have two tables, employees and employees_projects<\/code>. The latter table references each employee with their corresponding projects. Run SELECT *<\/code> from each table to take a look!<\/p>\n\n\n\n
Then, you could write a query like this:<\/p>\n\n\n
SELECT<\/span> e.first_name, e.last_name, e.salary\nFROM<\/span> employees e\nWHERE<\/span> e.id NOT<\/span> IN<\/span> (\n\u00a0 SELECT<\/span> ep.employee_id\n\u00a0 FROM<\/span> employees_projects ep\n\u00a0 WHERE<\/span> e.id = ep.employee_id\n);<\/code><\/span>Code language:<\/span> SQL (Structured Query Language)<\/span> (<\/span>sql<\/span>)<\/span><\/small><\/pre>\n\n\n
This query gets the employee data where the employee ID does not exist in the list of values of employee_id<\/code>‘s in the employees_projects<\/code> table. This is not a very efficient query if you have a table with million records. That’s because you wouldn’t need the employee_id<\/code>‘s values. Here you just need to check if they exist or not. In this case, you’ll need to return true values from the employees_projects<\/code> table, which can be used by SELECT<\/code> 1. You’ll also need to use the keyword EXISTS<\/code> to check the existence of these true values. For our query, we would negate the existence using the NOT<\/code> operator.<\/p>\n\n\n
SELECT<\/span> e.first_name, e.last_name, e.salary\nFROM<\/span> employees e\nWHERE<\/span> NOT<\/span> EXISTS<\/span> (\n\u00a0 SELECT<\/span> 1<\/span>\n\u00a0 FROM<\/span> employees_projects ep\n\u00a0 WHERE<\/span> e.id = ep.employee_id\n);<\/code><\/span>Code language:<\/span> SQL (Structured Query Language)<\/span> (<\/span>sql<\/span>)<\/span><\/small><\/pre>\n\n\n
The previous two queries provide the same output:<\/p>\n\n\n
\n
\"\"\/
Employee not present in the employees_projects<\/code> table.<\/figcaption><\/figure>\n<\/div>\n\n\n
As you can see, John Mills<\/strong> is the only employee who doesn’t have an ongoing project to work on.<\/p>\n\n\n\n
Self joins<\/strong><\/h2>\n\n\n\n
A self-join is used to join columns from the same table.<\/p>\n\n\n\n
Let\u2019s assume you want to retrieve employees who share the same last name:<\/p>\n\n\n
SELECT<\/span> e1.id, e1.first_name, e2.last_name\nFROM<\/span> employees e1, employees e2\nWHERE<\/span> e1.id <> e2.id\nAND<\/span> e1.last_name = e2.last_name;<\/code><\/span>Code language:<\/span> SQL (Structured Query Language)<\/span> (<\/span>sql<\/span>)<\/span><\/small><\/pre>\n\n\n
The FROM<\/code> clause is followed by two tables; the same employees table. There are two conditions in the WHERE<\/code> clause; ids are not the same, and the last names are the same.<\/p>\n\n\n
\n
\"\"\/
Employees sharing the same last name.<\/figcaption><\/figure>\n<\/div>\n\n\n
The above short-hand comma syntax is equivalent to an INNER JOIN<\/code> syntax (which we will illustrate later in this tutorial). So an equivalent query would be:<\/p>\n\n\n
SELECT<\/span> e1.id, e1.first_name, e2.last_name\nFROM<\/span> employees e1\nJOIN<\/span> employees e2 ON<\/span> e1.last_name = e2.last_name\nAND<\/span> e1.id <> e2.id;<\/code><\/span>Code language:<\/span> SQL (Structured Query Language)<\/span> (<\/span>sql<\/span>)<\/span><\/small><\/pre>\n\n\n
OUTER joins<\/strong><\/h2>\n\n\n\n
An OUTER<\/code> join is the FULL_OUTER_JOIN<\/code>. This join returns all records that meet the query condition, whether in the left table or in the right table.<\/p>\n\n\n\n
To make sure you understand OUTER<\/code> joins, experiment with the example below.<\/p>\n\n\n\n
Assume you have a record inserted in the employees table that has an empty value in a specific column like this:<\/p>\n\n\n
INSERT<\/span> INTO<\/span> employees(first_name, last_name, salary) VALUES<\/span>('Ibrahim'<\/span>, 'Saeid'<\/span>, 15000<\/span>);<\/code><\/span>Code language:<\/span> SQL (Structured Query Language)<\/span> (<\/span>sql<\/span>)<\/span><\/small><\/pre>\n\n\n
As you can see, Ibrahim Saied is added with a salary equal to 15000 but there was a little issue. He was not assigned to any department. In other words, the value of department_id<\/code> in his record is NULL<\/code>. Have a look at the new employees table running SELECT * FROM<\/strong> employees<\/code>:<\/p>\n\n\n
\n
\"\"\/
Updated employees list<\/figcaption><\/figure>\n<\/div>\n\n\n
Have a second look at the departments table, and you’ll see that there is a department that has no employees, which is Biz Dev, with an id of 4. That’s because department_id<\/code> has no value of 4 in the employees<\/code> table.<\/p>\n\n\n\n
Our task now is to list all employees’ names associated with their departments.<\/p>\n\n\n\n
You can accomplish different outcomes to solve this task. You need to know what requirements you really want to accomplish. In this section and the section that follows, we will discuss different scenarios.<\/p>\n\n\n\n
You need two tables; employees table to get employees names, and departments table to get departments names. That’s a given! but there are null values in each table!<\/p>\n\n\n\n
To list all employees with their departments, including both Ibrahim Saeid and the Biz Dev department, use FULL OUTER JOIN<\/code>:<\/p>\n\n\n
SELECT<\/span> e.first_name, e.last_name, e.salary, d.name AS<\/span> department\nFROM<\/span> employees e\nFULL<\/span> OUTER<\/span> JOIN<\/span> departments d ON<\/span> d.id = e.department_id;<\/code><\/span>Code language:<\/span> SQL (Structured Query Language)<\/span> (<\/span>sql<\/span>)<\/span><\/small><\/pre>\n\n
\n
\"\"\/
List of employees including their departments.<\/figcaption><\/figure>\n<\/div>\n\n\n
In the next section, we will discuss more scenarios.<\/p>\n\n\n\n
INNER joins<\/strong><\/h2>\n\n\n\n
An INNER<\/code> join is the conventional SQL join. This join statement joins two tables that match on both sides, the left and right tables.<\/p>\n\n\n\n
An INNER<\/code> join has 3 types:<\/p>\n\n\n\n
    \n
  • INNER JOIN<\/code> which returns all records that match the values in both tables<\/li>\n\n\n\n
  • LEFT INNER JOIN<\/code> which returns all records that meet the query condition in the left table, but doesn’t meet that condition in the right table<\/li>\n\n\n\n
  • RIGHT INNER JOIN<\/code> which is the reverse of the LEFT OUTER JOIN<\/code><\/li>\n<\/ul>\n\n\n\n
    \n
    \ud83d\udca1You can omit the INNER<\/code> keyword from each join statement.<\/p>\n<\/blockquote>\n\n\n\n
    So in the OUTER<\/code> join query example mentioned in the last section, we saw that both null values, from the employees table and the departments table, were returned. For the INNER JOIN<\/code>, the case is the reverse. You’ll get records that exist in both<\/em> tables:<\/p>\n\n\n
    SELECT<\/span> e.first_name, e.last_name, e.salary, d.name AS<\/span> department\nFROM<\/span> employees e\nJOIN<\/span> departments d ON<\/span> d.id = e.department_id;<\/code><\/span>Code language:<\/span> SQL (Structured Query Language)<\/span> (<\/span>sql<\/span>)<\/span><\/small><\/pre>\n\n
    \n
    \"\"\/
    Employees existing in both tables.<\/figcaption><\/figure>\n<\/div>\n\n\n
    As you can see, there are no null values that match the join condition.<\/p>\n\n\n\n
    To list all employees with their departments, including Ibrahim Saeid, who is not in a department:<\/p>\n\n\n
    SELECT<\/span> e.first_name, e.last_name, e.salary, d.name AS<\/span> department\nFROM<\/span> employees e\nLEFT<\/span> JOIN<\/span> departments d ON<\/span> d.id = e.department_id;<\/code><\/span>Code language:<\/span> SQL (Structured Query Language)<\/span> (<\/span>sql<\/span>)<\/span><\/small><\/pre>\n\n\n
    Here, we used LEFT JOIN<\/code> because we started selecting the employees table. So the output would be:<\/p>\n\n\n
    \n
    \"\"\/
    List of employees with and without a department.<\/figcaption><\/figure>\n<\/div>\n\n\n
    To list employees with their departments, including<\/em> the Biz Dev department, which has no employees, just change the previous query from LEFT JOIN<\/code> to RIGHT JOIN<\/code>:<\/p>\n\n\n
    SELECT<\/span> e.first_name, e.last_name, e.salary, d.name AS<\/span> department\nFROM<\/span> employees e\nRIGHT<\/span> JOIN<\/span> departments d ON<\/span> d.id = e.department_id;<\/code><\/span>Code language:<\/span> SQL (Structured Query Language)<\/span> (<\/span>sql<\/span>)<\/span><\/small><\/pre>\n\n
    \n
    \"\"\/
    List of employees with their departments<\/figcaption><\/figure>\n<\/div>\n\n\n
    Run the sandbox below to test the new queries learned in this article:<\/p>\n\n\n\n\t