Buy

Raw SQL Queries

All this Doctrine entity object stuff and DQL queries are really great. But if you ever feel overwhelmed with all of this or need write a really complex query, you can always fall back to using raw SQL. Seriously, this is huge. When Doctrine scares you, you are totally free to run away and use plain SQL. I won't judge you - just get that feature finished and launch it already. If a tool isn't helping you, don't use it.

Put on your DBA hat and let's write some SQL!

Open up FortuneCookieRepository. This is where we built a query that selected a sum, an average and the category name.

The DBAL Connection

The most important object in Doctrine is.... the entity manager! But that's just a puppet for the real guy in charge: the connection! Grab it by getting the entity manager and calling getConnection(). Let's var_dump() this:

32 lines src/AppBundle/Entity/FortuneCookieRepository.php
... lines 1 - 14
public function countNumberPrintedForCategory(Category $category)
{
$conn = $this->getEntityManager()
->getConnection();
var_dump($conn);die;
... lines 21 - 28
}
... lines 30 - 32

Head to the browser and click into one of the category pages. There's our beautiful var_dump(). Hey! Look at the class name:

Doctrine\DBAL\Connection

Fun fact! The Doctrine library is actually multiple libraries put together. The two parts we care about are the ORM and the DBAL. The ORM is what does all the magic mapping of data onto objects. The DBAL - or database abstraction layer - can be used completely independent of the ORM. It's basically a wrapper around PDO. Said in a less boring way, it's a library for executing database queries.

So this DBAL Connection objects is our key to running raw database queries. Google for "Doctrine DBAL Query" so we can follow its docs. Find the Data Retrieval And Manipulaton section. Scroll down a little to a good example:

$sql = "SELECT * FROM articles WHERE id = ?";
$stmt = $conn->prepare($sql);
$stmt->bindValue(1, $id);
$stmt->execute();

This DBAL library is a really light wrapper around PHP's PDO. So if you've used that before, you'll like this. But if not, it's like 3 steps, so stick with me.

Making a Raw SQL Query

Back in FortuneCookieRepository, let's write some simple SQL to test with:

SELECT * FROM fortune_cookie;

When you use the DBAL, there are no entities and it doesn't know about any of our Doctrine annotations. Yep, we're talking to the raw tables and columns. So I used fortune_cookies because that's the name of the actual table in the database.

Next, we'll use the SQL to get a statement. So:

$stmt = $conn->prepare($sql);

And then we can execute() that, which runs the query but doesn't give you the result. To get that, call $stmt->fetchAll() and var_dump() that:

35 lines src/AppBundle/Entity/FortuneCookieRepository.php
... lines 1 - 14
public function countNumberPrintedForCategory(Category $category)
{
$conn = $this->getEntityManager()
->getConnection();
$sql = 'SELECT * FROM fortune_cookie';
$stmt = $conn->prepare($sql);
$stmt->execute();
var_dump($stmt->fetchAll());die;
... lines 24 - 31
}
... lines 33 - 35

Try it! And there it is: exactly what you'd expect with no effort at all. It's literally the results - in array format - from the raw SQL query. Doctrine isn't trying to hide this feature from you - just grab the Connection object and you're dangerous.

Prepared Statements

The query we made with the query builder is a bit more complex. Could we replacle that with raw SQL? Sure! Well there's not really a good reason to do this, since it's built and working. But let's prove we can do it!

Let's grab the "select" part of the query and stick that in our query. I hate long lines, so let's use multiple. Piece by piece, add the other query parts. The FROM is fortune_cookie fc. Add the INNER JOIN to category ON cat.id = fc.category_id. And since we're in DBAL land, we don't have any of our annotation mapping configuration, so we have to tell it exactly how to join - it's just raw SQL. And for the same reason, we're using the real column names, like category_id.

Add a single WHERE of fc.category_id = :category. That's some good-old-fashioned boring SQL. I love it! The only thing we still need to do is fill in the :category placeholder. Even though we're using the DBAL, we still don't concatenate strings in our queries, unless you love SQL attacks or prefer to live dangerously. Are you feeling lucky, punk?

Ahem. To give :category a value, just pass an array to execute() and pass it a category key assigned to the id. Ok, done! Let's dump this!

40 lines src/AppBundle/Entity/FortuneCookieRepository.php
... lines 1 - 14
public function countNumberPrintedForCategory(Category $category)
{
$conn = $this->getEntityManager()
->getConnection();
$sql = '
SELECT SUM(fc.numberPrinted) as fortunesPrinted, AVG(fc.numberPrinted) as fortunesAverage, cat.name
FROM fortune_cookie fc
INNER JOIN category cat ON cat.id = fc.category_id
WHERE fc.category_id = :category
';
$stmt = $conn->prepare($sql);
$stmt->execute(array('category' => $category->getId()));
var_dump($stmt->fetchAll());die;
... lines 29 - 36
}
... lines 38 - 40

Boom! That's exactly what I was hoping for.

Using fetch() to get back the First Row

Since our SQL gives us just one row, it'd be awesome to get just its columns, instead of an array with one result. Just use fetch()!

40 lines src/AppBundle/Entity/FortuneCookieRepository.php
... lines 1 - 14
public function countNumberPrintedForCategory(Category $category)
{
$conn = $this->getEntityManager()
->getConnection();
$sql = '
SELECT SUM(fc.numberPrinted) as fortunesPrinted, AVG(fc.numberPrinted) as fortunesAverage, cat.name
FROM fortune_cookie fc
INNER JOIN category cat ON cat.id = fc.category_id
WHERE fc.category_id = :category
';
$stmt = $conn->prepare($sql);
$stmt->execute(array('category' => $category->getId()));
var_dump($stmt->fetch());die;
... lines 29 - 36
}
... lines 38 - 40

And now, this is exactly what our query builder gave us before. So get rid of the die() statement and return the fetch() line:

41 lines src/AppBundle/Entity/FortuneCookieRepository.php
... lines 1 - 14
public function countNumberPrintedForCategory(Category $category)
{
$conn = $this->getEntityManager()
->getConnection();
$sql = '
SELECT SUM(fc.numberPrinted) as fortunesPrinted, AVG(fc.numberPrinted) as fortunesAverage, cat.name
FROM fortune_cookie fc
INNER JOIN category cat ON cat.id = fc.category_id
WHERE fc.category_id = :category
';
$stmt = $conn->prepare($sql);
$stmt->execute(array('category' => $category->getId()));
return $stmt->fetch();
... lines 30 - 37
}
... lines 39 - 41

Just let the old code sit down there. Refresh! And we're prefectly back to normal. Man, that was kinda easy. So if Doctrine ever looks hard or you're still learning it, totally use SQL. It's no big deal.

Native Queries?

One slightly confusing thing is that if you google for "doctrine raw sql", you'll find a different solution - something called NativeQuery. It sort of looks the same, just with some different function names. But there's this ResultSetMapping thing. Huh. This NativeQuery thing allows you to run a raw SQL query and then map that back to an object. That's pretty neat I guess. But for me, if I'm writing some custom SQL, I'm fine just getting back an array of data. I can deal with that. The ResultSetMapping confuses me, and probably isn't worth the effort. But it's there if you want to geek out on it.

Leave a comment!

  • 2017-02-16 weaverryan

    Hey Zuhayer Tahir!

    Hmm, can you tell me one more thing? Can you describe exactly what you're trying to accomplish (I mean, what the page looks like that you're trying to build)? Are you building a page where you show the most recent message for all threads? Or are you building a page where you are showing all messages for a single thread, in a certain order? I might be able to help if I have the end-goal in mind :).

    Cheers!

  • 2017-02-15 Zuhayer Tahir

    The project is old And I cannot alter its major structure.

    The basic query is as follows:

    select * from ps_message p1 GROUP BY p1.thread_id

    currently it shows the msg 'first' my requirement is that it show the msg 'last'.

    id msg date
    1 first 2017-02-11 19:43:16
    2 b 2017-02-11 19:43:16
    3 c 2017-02-11 19:43:16
    4 d 2017-02-11 19:43:16
    5 e 2017-02-11 19:43:16
    6 Last 2017-02-11 19:43:16

    The ResultSetMapping() and Max() parts are used to achieve this result. I hope this simplifies it :)

  • 2017-02-13 weaverryan

    Hi Zuhayer Tahir!

    I can... kind of help you :). First, unfortunately, I never use the ResultSetMapping, so I'm not sure exactly what the problem is here. The reason I don't use it is that I think it's almost never necessary: I make 99% of my queries using the normal, query builder or DQL. And for that last 1% of really complex queries, I use SQL, but don't map it to objects (this is ok!).

    In this case, it looks like you're working with a bit of a complex, possibly legacy database structure. Even though I can't help, I would either continue to try with the ResultSetMapping, or try to simplify the database structure itself. It looks like you want to SELECT all messages joined to the original message from the same thread... in a more perfect world (if I understand things correctly), I would give each ps_message a foreign key to the original message in its thread (e.g. original_thread_message_id). Then, you could use a simple join instead of needing the MAX stuff.

    Sorry I can't be more help! Good luck!

  • 2017-02-11 Zuhayer Tahir

    Hi Ryan,

    I have this query (which is working fine).


    $sql = 'select * from ps_message p1 INNER JOIN (SELECT MAX(p2.id) as max_id FROM ps_message p2 GROUP BY p2.thread_id) as max_table on p1.id = max_table.max_id';

    I want to convert it for my Symfony2 project.

    I have tried building the DQL query using createQueryBuilder() and Native Query with ResultSetMapping()/ResultSetMappingBuilder(), but no success yet.

    P.s. With ResultSetMapping() i am unable to get the relation fields working. I have tried tinkering with addJoinedEntityResult(), etc..:


    $rsm = new ResultSetMapping();
    $rsm->addEntityResult('BaseBundle:PsMessage', 'p1');
    $rsm->addFieldResult('p1', 'id', 'id');
    $rsm->addFieldResult('p1', 'message_text', 'messageText');
    $rsm->addFieldResult('p1', 'message_type', 'messageType');
    $rsm->addFieldResult('p1', 'receiver_message_status', 'receiverMessageStatus');
    $rsm->addFieldResult('p1', 'sender_message_status', 'senderMessageStatus');
    $rsm->addFieldResult('p1', 'created_date', 'createdDate');
    $rsm->addFieldResult('p1', 'modified_date', 'modifiedDate');

    $rsm->addFieldResult('t1', 'thread_id', 'thread_id'); //Error
    $rsm->addFieldResult('u', 'sender_id', 'sender'); //Error
    $rsm->addFieldResult('t', 'thread_id', 'thread'); //Error
    $rsm->addFieldResult('p2', 'thread_id', 'thread'); //Error

    $sql = 'select * from ps_message p1 INNER JOIN (SELECT MAX(p2.id) as max_id FROM ps_message p2 GROUP BY p2.thread_id) as max_table on p1.id = max_table.max_id';
    $query = $em->createNativeQuery($sql, $rsm);

    Could you kindly guide me?

  • 2015-10-15 Daniel

    Wrote you from my business mail (daniel@timemanagerweb.co).

  • 2015-10-15 weaverryan

    We do them in the US too, both in site and publicly from time to time (I give the US trainings). Send me a message - ryan@knplabs.com if you're interested :)

  • 2015-10-15 Daniel

    Thanks Ryan. I am seriously interested on taking the on premise training. Is it in the US or only in France?

  • 2015-10-14 weaverryan

    Hey Daniel!

    So very nice of you to day - I'm thrilled you're coming from raw php to Symfony - these are skill that you will not regret :).

    I am not a Postgres expert (actually, I completely think it's superior to MySQL, but still have not used it extensively), but my guess is that yes, this is probably do to a difference in the databases themselves. In the case of the query builder, usually Doctrine abstracts things away and makes queries that work on all database systems. But, as you can see, it *is* generating valid PgSQL code, but yes, it looks like you must need a GROUP BY in order to do the sum and/or average.

    So, I wouldn't worry about it too much - I think you came to the right conclusion and are understanding things correctly.

    Cheers!

  • 2015-10-13 Daniel

    Hi Ryan! Me again. Really this site for me has become an absolutely essential tool for migrating from raw php to symfony, I've been devouring all tutorials! Not long ago I decided to move to Postgres, I fall in love with the jsonb data type and many other advanced things that I miss in MySql, so, I'm following this tutorial using Postgres 9.4. In this screen cast and in a previous one I ran into an PDO error, here goes as:

    An exception occurred while executing '
    SELECT SUM(fc.numberPrinted) as fortunesPrinted, AVG(fc.numberPrinted) as fortunesAverage, cat.name
    FROM fortune_cookie fc
    INNER JOIN category cat ON cat.id = fc.category_id
    WHERE fc.category_id = :category
    ' with params [3]:

    But, if I do something like this (I did as the error suggested), the error goes away:

    SELECT SUM(fc.numberPrinted) as fortunesPrinted, AVG(fc.numberPrinted) as fortunesAverage, cat.name
    FROM fortune_cookie fc
    INNER JOIN category cat ON cat.id = fc.category_id
    WHERE fc.category_id = 3
    group by cat.name

    May it be a difference with DB engine or something on Doctrine. I got something very similar on the screen cast before this one, so I changed it to:

    return $this->createQueryBuilder('fc')
    ->andWhere('fc.category = :category')
    ->setParameter('category', $category)
    ->innerJoin('fc.category', 'cat')
    ->select('SUM(fc.numberPrinted) as fortunesPrinted, AVG(fc.numberPrinted) as fortunesAverage, cat.name')
    ->addGroupBy('cat.name')
    ->getQuery()
    ->getOneOrNullResult();

    Cheers!