The most performant way to get a set of the ID is already well known:

// ids: [001000000000001AAA, 001000000000002BBB, ...]
Set<Id> ids = new Map<Id,SObject>(records).keySet();

We can also build sets from ANY string field, in a one-liner without a loop:

// names: [Matt, Neil, Chatter, Autoproc, ...]
Set<String> names = new Map<String,SObject>([
    SELECT Name Id
    FROM User
    GROUP BY Name

How it works

The first thing to note is that we are using an Aggregate Query and not just ordinary SOQL. We combine special behaviours available to aggregate queries to build the set:

Step 1 - GROUP BY to collect unique values

Exactly per the documentation, you can use the GROUP BY option in a SOQL query to avoid iterating through individual query results. That is, you specify a group of records instead of processing many individual records.

Step 2 - Alias the field as Id

The default identifier for an aggregated field is expr0 but we will alias the result as Id. It doesn't matter what the field contains, so long as it's a string. We will exploit this special alias in the next step.

Step 3 - AggregateResult inherits from sObject

The methods available on sObject are also available on AggregateResult. This also applies to Lists and Maps; those same methods can be called on collections of aggregate results. Here is another example using keySet()

// emails: [,, ...]
Set<String> emails = new Map<String,SObject>([
    SELECT Email Id
    FROM Contact WHERE Email != null
    GROUP BY Email

this is a superb trick