Dummy Variables and Column Transformers

Activity 3.1

Fill vs Color for Scatterplots

Code

(
    ggplot(df, mapping = aes(x = "horsepower", y = "mpg", fill = "origin")) +
    geom_point() +
    theme_bw() +
    labs(x = "Horsepower",
         y = "Miles Per Gallon",
         color = "Country of Manufacture")
    )

Code

(
    ggplot(df, mapping = aes(x = "horsepower", y = "mpg", color = "origin")) +
    geom_point() +
    theme_bw() +
    labs(x = "Horsepower",
         y = "Miles Per Gallon",
         color = "Country of Manufacture")
    )

Nicely Formatted Axis Labels

Code

(
    ggplot(df, mapping = aes(x = "model year",
                             y = "mpg",
                             color = "origin")
    ) +
    geom_point() +
    theme_bw() +
    labs(x = "Year of Manufacture",
         y = "Miles Per Gallon",
         color = "Country of Manufacture") 
)

Code

(
    ggplot(df, mapping = aes(x = "model year",
                             y = "mpg",
                             color = "origin")
    ) +
    geom_point() +
    theme_bw() +
    labs(x = "Year of Manufacture",
         y = "Miles Per Gallon",
         color = "Country of Manufacture") +
    scale_x_continuous(
        breaks = list(range(1970, 1984, 2)
        )
        )
)

Displaying Every Point

Code

(
    ggplot(df, mapping = aes(x = "model year",
                             y = "mpg",
                             color = "origin")
    ) +
    geom_jitter() +
    theme_bw() +
    labs(x = "Year of Manufacture",
         y = "Miles Per Gallon",
         color = "Country of Manufacture") +
    scale_x_continuous(
        breaks = list(range(1970, 1984, 2)
        )
        )
)

Redundant Legends

Removing Your Legend

Code

(
    ggplot(df, mapping = aes(x = "model year",
                             y = "mpg",
                             color = "origin")
    ) +
    geom_jitter() +
    facet_wrap("origin") +
    theme_bw() +
    labs(x = "Year of Manufacture",
         y = "Miles Per Gallon",
         color = "Country of Manufacture") +
    scale_x_continuous(
        breaks = list(range(1970, 1985, 5)
        )
        ) + 
    theme(legend_position = "none")
)

Adding Descriptive Labels to Facets

Code

(
    ggplot(df, aes(x = 'horsepower', y = 'mpg', color = 'origin'))
    + geom_point()
    + facet_wrap('~ cylinders', labeller = labeller(cylinders=lambda x: f'{x} cylinders'))
    + labs(title = 'Horsepower vs. Fuel Efficiency by Origin and Cylinders',
    x = 'Horsepower', 
    y = 'Fuel Efficiency (MPG)', color = 'Origin')
    + theme_bw()
)

`geom_line()`

Code

(
    ggplot(df, mapping = aes(x = "model year",
                             y = "mpg",
                             color = "origin")
    ) +
    geom_line() +
    theme_bw() +
    labs(x = "Year of Manufacture",
         y = "Miles Per Gallon",
         color = "Country of Manufacture") +
    scale_x_continuous(
        breaks = list(range(1970, 1985, 5)
        )
        )
)

Code

year_means = (
    df
    .groupby(["model year", "origin"])["mpg"]
    .mean()
    .reset_index()
)

(
    ggplot(year_means, mapping = aes(x = "model year",
                                      y = "mpg",
                                      color = "origin")
    ) +
    geom_line() +
    theme_bw() +
    labs(x = "Year of Manufacture",
         y = "Miles Per Gallon",
         color = "Country of Manufacture") +
    scale_x_continuous(
        breaks = list(range(1970, 1985, 5)
        )
        )
)

The story last week…

Distances

We measure similarity between observations by calculating distances.

Euclidean distance: sum of squared differences, then square root
Manhattan distance: sum of absolute differences

scikit-learn

Use the pairwise_distances() function to get back a 2D numpy array of distances.

Scaling

It is important that all our features be on the same scale for distances to be meaningful.

Standardize: Subtract the mean (of the column) and divide by the standard deviation (of the column).
MinMax: Subtract the minimum value, divide by the range.

scikit-learn

Follow the specify - fit - transform code structure. In the specify step, you should use the StandardScaler() or MinMaxScaler() functions.

Recall: AMES Housing data

df = pd.read_table("data/housing.tsv")
df.head()

   Order        PID  MS SubClass  ... Sale Type  Sale Condition  SalePrice
0      1  526301100           20  ...       WD           Normal     215000
1      2  526350040           20  ...       WD           Normal     105000
2      3  526351010           20  ...       WD           Normal     172000
3      4  526353030           20  ...       WD           Normal     244000
4      5  527105010           60  ...       WD           Normal     189900

[5 rows x 82 columns]

Distances and Categorical Variables

What about categorical variables?

Suppose we want to include the variable Bldg Type in our distance calculation…

df["Bldg Type"].value_counts()

Bldg Type
1Fam      2425
TwnhsE     233
Duplex     109
Twnhs      101
2fmCon      62
Name: count, dtype: int64

Then we need a way to calculate $(\texttt{1Fam} - \texttt{Twnhs} )^ 2$.

Converting to Binary

Let’s instead think about a variable that summarizes whether an observation is a single family home or not.

df["is_single_fam"] = df["Bldg Type"] == "1Fam"
df["is_single_fam"].value_counts()

is_single_fam
True     2425
False     505
Name: count, dtype: int64

What does a value of True represent? False?

Dummy Variables

When we transform a variable into binary (True / False), we call this variable a dummy variable or we say the variable has been one-hot-encoded.

Remember that that computers interpret logical values (True / False) the same as 1 / 0:

df["is_single_fam"] = df["is_single_fam"].astype("int")
df["is_single_fam"].value_counts()

is_single_fam
1    2425
0     505
Name: count, dtype: int64

Now we can do math!

Specify

from sklearn.preprocessing import StandardScaler
from sklearn.metrics import pairwise_distances

scaler = StandardScaler()

Fit

df_orig = df[['Gr Liv Area', 'Bedroom AbvGr', 'is_single_fam']]
scaler.fit(df_orig)

StandardScaler()

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Transform

df_scaled = scaler.transform(df_orig)

Calculating Distances

dists = pairwise_distances(df_scaled[[1707]], df_scaled)
best = (
  dists
  .argsort()
  .flatten()
  [0:6]
  )
df_orig.iloc[best]

      Gr Liv Area  Bedroom AbvGr  is_single_fam
1707         2956              5              1
160          2978              5              1
909          3082              5              1
1288         2792              5              1
2350         2784              5              1
253          3222              5              1

Looking back…

Where have you seen one-hot-encoded variables already?

Let’s reset the dataset now…

df = pd.read_table("data/housing.tsv")

Dummifying Variables

What if we don’t just want to study is_single_fam, but rather, all categories of the Bldg Type variable?
In principle, we just make dummy variables for each category: is_single_fam, is_twnhse, etc.
Each category becomes one column, with 0’s and 1’s to show if the observation in that row matches that category.
That sounds pretty tedious, especially if you have a lot of categories…
Luckily, we have shortcuts in both pandas and sklearn!

Dummifying in Pandas

pd.get_dummies(df[["Bldg Type"]])

      Bldg Type_1Fam  Bldg Type_2fmCon  ...  Bldg Type_Twnhs  Bldg Type_TwnhsE
0               True             False  ...            False             False
1               True             False  ...            False             False
2               True             False  ...            False             False
3               True             False  ...            False             False
4               True             False  ...            False             False
...              ...               ...  ...              ...               ...
2925            True             False  ...            False             False
2926            True             False  ...            False             False
2927            True             False  ...            False             False
2928            True             False  ...            False             False
2929            True             False  ...            False             False

[2930 rows x 5 columns]

Dummifying in Pandas

      Bldg Type_1Fam  Bldg Type_2fmCon  ...  Bldg Type_Twnhs  Bldg Type_TwnhsE
0               True             False  ...            False             False
1               True             False  ...            False             False
2               True             False  ...            False             False
3               True             False  ...            False             False
4               True             False  ...            False             False
...              ...               ...  ...              ...               ...
2925            True             False  ...            False             False
2926            True             False  ...            False             False
2927            True             False  ...            False             False
2928            True             False  ...            False             False
2929            True             False  ...            False             False

[2930 rows x 5 columns]

Some things to notice here…

What is the naming convention for the new columns?
Does this change the original dataframe df? If not, what would you need to do to add this information back in?
What happens if you put the whole dataframe into the get_dummies function? What problems might arise from this?

Dummifying in sklearn

Specify

from sklearn.preprocessing import OneHotEncoder

encoder = OneHotEncoder()

Fit

encoder.fit(df[["Bldg Type"]])

OneHotEncoder()

Transform

df_bldg = encoder.transform(df[["Bldg Type"]])

Dummifying in sklearn

df_bldg

<Compressed Sparse Row sparse matrix of dtype 'float64'
    with 2930 stored elements and shape (2930, 5)>

df_bldg.todense()

matrix([[1., 0., 0., 0., 0.],
        [1., 0., 0., 0., 0.],
        [1., 0., 0., 0., 0.],
        ...,
        [1., 0., 0., 0., 0.],
        [1., 0., 0., 0., 0.],
        [1., 0., 0., 0., 0.]], shape=(2930, 5))

Things to notice:

What object type was the result?
Does this change the original dataframe df? If not, what would you need to do to add this information back in?
What pros and cons do you see for the pandas approach vs the sklearn approach?

Column Transformers

Preprocessing

So far, we have now seen two preprocessing steps that might need to happen to do an analysis of distances:

Scaling the quantitative variables
Dummifying the categorical variables

Preprocessing steps are things you do only to make the following analysis/visualization better.

This is not the same as data cleaning, where you make changes to fix the data (e.g., changing data types).
This is not the same as data wrangling, where you change the structure of the data (e.g., adding or deleting rows or columns).

Lecture 4.1 Quiz

Identify the following as cleaning, wrangling, or preprocessing:

Removing the $ symbol from a column and converting it to numeric.
Narrowing your data down to only first class Titanic passengers, because you are not studying the others.
Converting a Zip Code variable from numeric to categorical using .astype().
Creating a new column called n_investment that counts the number of people who invested in a project.
Log-transforming a column because it is very skewed.

Preprocessing in `sklearn`

Unlike cleaning and wrangling, the preprocessing steps are “temporary” changes to the dataframe.

It would be nice if we could trigger these changes as part of our analysis, instead of doing them “by hand”.
- This is why the specify - fit - transform process is useful!
- We will first specify all our preprocessing steps.
- Then we will fit the whole preprocess
- Then we will save the transform step for only when we need it.

Column Transformers – Specify

from sklearn.compose import make_column_transformer

features = ["Gr Liv Area", "Bedroom AbvGr", "Full Bath", "Half Bath", "Bldg Type", "Neighborhood"]

preproc = make_column_transformer(
  (OneHotEncoder(), ["Bldg Type", "Neighborhood"]),
    remainder = "passthrough")

Column Transformers – Fit

preproc.fit(df[features])

ColumnTransformer(remainder='passthrough',
                  transformers=[('onehotencoder', OneHotEncoder(),
                                 ['Bldg Type', 'Neighborhood'])])

Column Transformers – Transform

preproc.transform(df[features])

<Compressed Sparse Row sparse matrix of dtype 'float64'
    with 15717 stored elements and shape (2930, 37)>

Things to notice…

What submodule did we import make_column_transformer from?
What are the two arguments to the make_column_transformer() function? What object structures are they?
What happens if you fit and transform on the whole dataset, not just df[features]? Why might this be useful?

Lecture Activity 4.2

Try the following:

What happens if you change remainder = "passthrough" to remainder = "drop"?
What happens if you add the argument sparse_output = False to the OneHotEncoder() function?
What happens if you add this line before the transform step: preproc.set_output(transform = "pandas") (keep the sparse_output = False when you try this)

Multiple Preprocessing Steps

Why are column transformers so useful? We can do multiple preprocessing steps at once!

from sklearn.preprocessing import StandardScaler

preproc = make_column_transformer(
        (StandardScaler(), 
        ["Gr Liv Area", "Bedroom AbvGr", "Full Bath", "Half Bath"]),
        (OneHotEncoder(sparse_output = False), 
        ["Bldg Type", "Neighborhood"]),
        remainder = "drop")

Fit!

preproc.fit(df)

ColumnTransformer(transformers=[('standardscaler', StandardScaler(),
                                 ['Gr Liv Area', 'Bedroom AbvGr', 'Full Bath',
                                  'Half Bath']),
                                ('onehotencoder',
                                 OneHotEncoder(sparse_output=False),
                                 ['Bldg Type', 'Neighborhood'])])

preproc.set_output(transform = "pandas")

ColumnTransformer(transformers=[('standardscaler', StandardScaler(),
                                 ['Gr Liv Area', 'Bedroom AbvGr', 'Full Bath',
                                  'Half Bath']),
                                ('onehotencoder',
                                 OneHotEncoder(sparse_output=False),
                                 ['Bldg Type', 'Neighborhood'])])

Transform!

df_transformed = preproc.transform(df)
df_transformed

      standardscaler__Gr Liv Area  ...  onehotencoder__Neighborhood_Veenker
0                        0.309265  ...                                  0.0
1                       -1.194427  ...                                  0.0
2                       -0.337718  ...                                  0.0
3                        1.207523  ...                                  0.0
4                        0.255844  ...                                  0.0
...                           ...  ...                                  ...
2925                    -0.982723  ...                                  0.0
2926                    -1.182556  ...                                  0.0
2927                    -1.048015  ...                                  0.0
2928                    -0.219006  ...                                  0.0
2929                     0.989884  ...                                  0.0

[2930 rows x 37 columns]

Finding All Categorical Variables

What if we want to tell sklearn, “Please dummify every categorical variable.”? Use a selector instead of exact column names!

from sklearn.compose import make_column_selector

preproc = make_column_transformer(
    (StandardScaler(),  
     make_column_selector(dtype_include = np.number)
     ),
    (OneHotEncoder(sparse_output = False), 
     make_column_selector(dtype_include = object)
     ),
    remainder = "passthrough")

preproc.set_output(transform = "pandas")

ColumnTransformer(remainder='passthrough',
                  transformers=[('standardscaler', StandardScaler(),
                                 <sklearn.compose._column_transformer.make_column_selector object at 0x306de78f0>),
                                ('onehotencoder',
                                 OneHotEncoder(sparse_output=False),
                                 <sklearn.compose._column_transformer.make_column_selector object at 0x3073b34a0>)])

Fit AND Transform!

preproc.fit_transform(df[features])

      standardscaler__Gr Liv Area  ...  onehotencoder__Neighborhood_Veenker
0                        0.309265  ...                                  0.0
1                       -1.194427  ...                                  0.0
2                       -0.337718  ...                                  0.0
3                        1.207523  ...                                  0.0
4                        0.255844  ...                                  0.0
...                           ...  ...                                  ...
2925                    -0.982723  ...                                  0.0
2926                    -1.182556  ...                                  0.0
2927                    -1.048015  ...                                  0.0
2928                    -0.219006  ...                                  0.0
2929                     0.989884  ...                                  0.0

[2930 rows x 37 columns]

Think about it

What are the advantages of using a selector?
What are the possible disadvantages of using a selector?
Does the order matter when using selectors? Try switching the steps and see what happens!

Takeaways

We dummify or one-hot-encode categorical variables to make them numbers.
We can do this with pd.get_dummies() or with OneHotEncoder() from sklearn.
Column Transformers let us apply multiple preprocessing steps at the same time.
- Think about which variables you want to apply the steps to
- Think about options for the steps, like sparseness
- Think about passthrough in your transformer

	categories categories: 'auto' or a list of array-like, default='auto' Categories (unique values) per feature: - 'auto' : Determine categories automatically from the training data. - list : ``categories[i]`` holds the categories expected in the ith column. The passed categories should not mix strings and numeric values within a single feature, and should be sorted in case of numeric values. The used categories can be found in the ``categories_`` attribute. .. versionadded:: 0.20	'auto'
	drop drop: {'first', 'if_binary'} or an array-like of shape (n_features,), default=None Specifies a methodology to use to drop one of the categories per feature. This is useful in situations where perfectly collinear features cause problems, such as when feeding the resulting data into an unregularized linear regression model. However, dropping one category breaks the symmetry of the original representation and can therefore induce a bias in downstream models, for instance for penalized linear classification or regression models. - None : retain all features (the default). - 'first' : drop the first category in each feature. If only one category is present, the feature will be dropped entirely. - 'if_binary' : drop the first category in each feature with two categories. Features with 1 or more than 2 categories are left intact. - array : ``drop[i]`` is the category in feature ``X[:, i]`` that should be dropped. When `max_categories` or `min_frequency` is configured to group infrequent categories, the dropping behavior is handled after the grouping. .. versionadded:: 0.21 The parameter `drop` was added in 0.21. .. versionchanged:: 0.23 The option `drop='if_binary'` was added in 0.23. .. versionchanged:: 1.1 Support for dropping infrequent categories.	None
	sparse_output sparse_output: bool, default=True When ``True``, it returns a :class:`scipy.sparse.csr_matrix`, i.e. a sparse matrix in "Compressed Sparse Row" (CSR) format. .. versionadded:: 1.2 `sparse` was renamed to `sparse_output`	True
	dtype dtype: number type, default=np.float64 Desired dtype of output.	<class 'numpy.float64'>
	handle_unknown handle_unknown: {'error', 'ignore', 'infrequent_if_exist', 'warn'}, default='error' Specifies the way unknown categories are handled during :meth:`transform`. - 'error' : Raise an error if an unknown category is present during transform. - 'ignore' : When an unknown category is encountered during transform, the resulting one-hot encoded columns for this feature will be all zeros. In the inverse transform, an unknown category will be denoted as None. - 'infrequent_if_exist' : When an unknown category is encountered during transform, the resulting one-hot encoded columns for this feature will map to the infrequent category if it exists. The infrequent category will be mapped to the last position in the encoding. During inverse transform, an unknown category will be mapped to the category denoted `'infrequent'` if it exists. If the `'infrequent'` category does not exist, then :meth:`transform` and :meth:`inverse_transform` will handle an unknown category as with `handle_unknown='ignore'`. Infrequent categories exist based on `min_frequency` and `max_categories`. Read more in the :ref:`User Guide `. - 'warn' : When an unknown category is encountered during transform a warning is issued, and the encoding then proceeds as described for `handle_unknown="infrequent_if_exist"`. .. versionchanged:: 1.1 `'infrequent_if_exist'` was added to automatically handle unknown categories and infrequent categories. .. versionadded:: 1.6 The option `"warn"` was added in 1.6.	'error'
	min_frequency min_frequency: int or float, default=None Specifies the minimum frequency below which a category will be considered infrequent. - If `int`, categories with a smaller cardinality will be considered infrequent. - If `float`, categories with a smaller cardinality than `min_frequency * n_samples` will be considered infrequent. .. versionadded:: 1.1 Read more in the :ref:`User Guide `.	None
	max_categories max_categories: int, default=None Specifies an upper limit to the number of output features for each input feature when considering infrequent categories. If there are infrequent categories, `max_categories` includes the category representing the infrequent categories along with the frequent categories. If `None`, there is no limit to the number of output features. .. versionadded:: 1.1 Read more in the :ref:`User Guide `.	None
	feature_name_combiner feature_name_combiner: "concat" or callable, default="concat" Callable with signature `def callable(input_feature, category)` that returns a string. This is used to create feature names to be returned by :meth:`get_feature_names_out`. `"concat"` concatenates encoded feature name and category with `feature + "_" + str(category)`.E.g. feature X with values 1, 6, 7 create feature names `X_1, X_6, X_7`. .. versionadded:: 1.3	'concat'

	transformers transformers: list of tuples List of (name, transformer, columns) tuples specifying the transformer objects to be applied to subsets of the data. name : str Like in Pipeline and FeatureUnion, this allows the transformer and its parameters to be set using ``set_params`` and searched in grid search. transformer : {'drop', 'passthrough'} or estimator Estimator must support :term:`fit` and :term:`transform`. Special-cased strings 'drop' and 'passthrough' are accepted as well, to indicate to drop the columns or to pass them through untransformed, respectively. columns : str, array-like of str, int, array-like of int, array-like of bool, slice or callable Indexes the data on its second axis. Integers are interpreted as positional columns, while strings can reference DataFrame columns by name. A scalar string or int should be used where ``transformer`` expects X to be a 1d array-like (vector), otherwise a 2d array will be passed to the transformer. A callable is passed the input data `X` and can return any of the above. To select multiple columns by name or dtype, you can use :obj:`make_column_selector`.	[('onehotencoder', ...)]
	remainder remainder: {'drop', 'passthrough'} or estimator, default='drop' By default, only the specified columns in `transformers` are transformed and combined in the output, and the non-specified columns are dropped. (default of ``'drop'``). By specifying ``remainder='passthrough'``, all remaining columns that were not specified in `transformers`, but present in the data passed to `fit` will be automatically passed through. This subset of columns is concatenated with the output of the transformers. For dataframes, extra columns not seen during `fit` will be excluded from the output of `transform`. By setting ``remainder`` to be an estimator, the remaining non-specified columns will use the ``remainder`` estimator. The estimator must support :term:`fit` and :term:`transform`. Note that using this feature requires that the DataFrame columns input at :term:`fit` and :term:`transform` have identical order.	'passthrough'
	sparse_threshold sparse_threshold: float, default=0.3 If the output of the different transformers contains sparse matrices, these will be stacked as a sparse matrix if the overall density is lower than this value. Use ``sparse_threshold=0`` to always return dense. When the transformed output consists of all dense data, the stacked result will be dense, and this keyword will be ignored.	0.3
	n_jobs n_jobs: int, default=None Number of jobs to run in parallel. ``None`` means 1 unless in a :obj:`joblib.parallel_backend` context. ``-1`` means using all processors. See :term:`Glossary ` for more details.	None
	transformer_weights transformer_weights: dict, default=None Multiplicative weights for features per transformer. The output of the transformer is multiplied by these weights. Keys are transformer names, values the weights.	None
	verbose verbose: bool, default=False If True, the time elapsed while fitting each transformer will be printed as it is completed.	False
	verbose_feature_names_out verbose_feature_names_out: bool, str or Callable[[str, str], str], default=True - If True, :meth:`ColumnTransformer.get_feature_names_out` will prefix all feature names with the name of the transformer that generated that feature. It is equivalent to setting `verbose_feature_names_out="{transformer_name}__{feature_name}"`. - If False, :meth:`ColumnTransformer.get_feature_names_out` will not prefix any feature names and will error if feature names are not unique. - If ``Callable[[str, str], str]``, :meth:`ColumnTransformer.get_feature_names_out` will rename all the features using the name of the transformer. The first argument of the callable is the transformer name and the second argument is the feature name. The returned string will be the new feature name. - If ``str``, it must be a string ready for formatting. The given string will be formatted using two field names: ``transformer_name`` and ``feature_name``. e.g. ``"{feature_name}__{transformer_name}"``. See :meth:`str.format` method from the standard library for more info. .. versionadded:: 1.0 .. versionchanged:: 1.6 `verbose_feature_names_out` can be a callable or a string to be formatted.	True
	force_int_remainder_cols force_int_remainder_cols: bool, default=False This parameter has no effect. .. note:: If you do not access the list of columns for the remainder columns in the `transformers_` fitted attribute, you do not need to set this parameter. .. versionadded:: 1.5 .. versionchanged:: 1.7 The default value for `force_int_remainder_cols` will change from `True` to `False` in version 1.7. .. deprecated:: 1.7 `force_int_remainder_cols` is deprecated and will be removed in 1.9.	'deprecated'

	categories categories: 'auto' or a list of array-like, default='auto' Categories (unique values) per feature: - 'auto' : Determine categories automatically from the training data. - list : ``categories[i]`` holds the categories expected in the ith column. The passed categories should not mix strings and numeric values within a single feature, and should be sorted in case of numeric values. The used categories can be found in the ``categories_`` attribute. .. versionadded:: 0.20	'auto'
	drop drop: {'first', 'if_binary'} or an array-like of shape (n_features,), default=None Specifies a methodology to use to drop one of the categories per feature. This is useful in situations where perfectly collinear features cause problems, such as when feeding the resulting data into an unregularized linear regression model. However, dropping one category breaks the symmetry of the original representation and can therefore induce a bias in downstream models, for instance for penalized linear classification or regression models. - None : retain all features (the default). - 'first' : drop the first category in each feature. If only one category is present, the feature will be dropped entirely. - 'if_binary' : drop the first category in each feature with two categories. Features with 1 or more than 2 categories are left intact. - array : ``drop[i]`` is the category in feature ``X[:, i]`` that should be dropped. When `max_categories` or `min_frequency` is configured to group infrequent categories, the dropping behavior is handled after the grouping. .. versionadded:: 0.21 The parameter `drop` was added in 0.21. .. versionchanged:: 0.23 The option `drop='if_binary'` was added in 0.23. .. versionchanged:: 1.1 Support for dropping infrequent categories.	None
	sparse_output sparse_output: bool, default=True When ``True``, it returns a :class:`scipy.sparse.csr_matrix`, i.e. a sparse matrix in "Compressed Sparse Row" (CSR) format. .. versionadded:: 1.2 `sparse` was renamed to `sparse_output`	True
	dtype dtype: number type, default=np.float64 Desired dtype of output.	<class 'numpy.float64'>
	handle_unknown handle_unknown: {'error', 'ignore', 'infrequent_if_exist', 'warn'}, default='error' Specifies the way unknown categories are handled during :meth:`transform`. - 'error' : Raise an error if an unknown category is present during transform. - 'ignore' : When an unknown category is encountered during transform, the resulting one-hot encoded columns for this feature will be all zeros. In the inverse transform, an unknown category will be denoted as None. - 'infrequent_if_exist' : When an unknown category is encountered during transform, the resulting one-hot encoded columns for this feature will map to the infrequent category if it exists. The infrequent category will be mapped to the last position in the encoding. During inverse transform, an unknown category will be mapped to the category denoted `'infrequent'` if it exists. If the `'infrequent'` category does not exist, then :meth:`transform` and :meth:`inverse_transform` will handle an unknown category as with `handle_unknown='ignore'`. Infrequent categories exist based on `min_frequency` and `max_categories`. Read more in the :ref:`User Guide `. - 'warn' : When an unknown category is encountered during transform a warning is issued, and the encoding then proceeds as described for `handle_unknown="infrequent_if_exist"`. .. versionchanged:: 1.1 `'infrequent_if_exist'` was added to automatically handle unknown categories and infrequent categories. .. versionadded:: 1.6 The option `"warn"` was added in 1.6.	'error'
	min_frequency min_frequency: int or float, default=None Specifies the minimum frequency below which a category will be considered infrequent. - If `int`, categories with a smaller cardinality will be considered infrequent. - If `float`, categories with a smaller cardinality than `min_frequency * n_samples` will be considered infrequent. .. versionadded:: 1.1 Read more in the :ref:`User Guide `.	None
	max_categories max_categories: int, default=None Specifies an upper limit to the number of output features for each input feature when considering infrequent categories. If there are infrequent categories, `max_categories` includes the category representing the infrequent categories along with the frequent categories. If `None`, there is no limit to the number of output features. .. versionadded:: 1.1 Read more in the :ref:`User Guide `.	None
	feature_name_combiner feature_name_combiner: "concat" or callable, default="concat" Callable with signature `def callable(input_feature, category)` that returns a string. This is used to create feature names to be returned by :meth:`get_feature_names_out`. `"concat"` concatenates encoded feature name and category with `feature + "_" + str(category)`.E.g. feature X with values 1, 6, 7 create feature names `X_1, X_6, X_7`. .. versionadded:: 1.3	'concat'

	transformers transformers: list of tuples List of (name, transformer, columns) tuples specifying the transformer objects to be applied to subsets of the data. name : str Like in Pipeline and FeatureUnion, this allows the transformer and its parameters to be set using ``set_params`` and searched in grid search. transformer : {'drop', 'passthrough'} or estimator Estimator must support :term:`fit` and :term:`transform`. Special-cased strings 'drop' and 'passthrough' are accepted as well, to indicate to drop the columns or to pass them through untransformed, respectively. columns : str, array-like of str, int, array-like of int, array-like of bool, slice or callable Indexes the data on its second axis. Integers are interpreted as positional columns, while strings can reference DataFrame columns by name. A scalar string or int should be used where ``transformer`` expects X to be a 1d array-like (vector), otherwise a 2d array will be passed to the transformer. A callable is passed the input data `X` and can return any of the above. To select multiple columns by name or dtype, you can use :obj:`make_column_selector`.	[('standardscaler', ...), ('onehotencoder', ...)]
	remainder remainder: {'drop', 'passthrough'} or estimator, default='drop' By default, only the specified columns in `transformers` are transformed and combined in the output, and the non-specified columns are dropped. (default of ``'drop'``). By specifying ``remainder='passthrough'``, all remaining columns that were not specified in `transformers`, but present in the data passed to `fit` will be automatically passed through. This subset of columns is concatenated with the output of the transformers. For dataframes, extra columns not seen during `fit` will be excluded from the output of `transform`. By setting ``remainder`` to be an estimator, the remaining non-specified columns will use the ``remainder`` estimator. The estimator must support :term:`fit` and :term:`transform`. Note that using this feature requires that the DataFrame columns input at :term:`fit` and :term:`transform` have identical order.	'drop'
	sparse_threshold sparse_threshold: float, default=0.3 If the output of the different transformers contains sparse matrices, these will be stacked as a sparse matrix if the overall density is lower than this value. Use ``sparse_threshold=0`` to always return dense. When the transformed output consists of all dense data, the stacked result will be dense, and this keyword will be ignored.	0.3
	n_jobs n_jobs: int, default=None Number of jobs to run in parallel. ``None`` means 1 unless in a :obj:`joblib.parallel_backend` context. ``-1`` means using all processors. See :term:`Glossary ` for more details.	None
	transformer_weights transformer_weights: dict, default=None Multiplicative weights for features per transformer. The output of the transformer is multiplied by these weights. Keys are transformer names, values the weights.	None
	verbose verbose: bool, default=False If True, the time elapsed while fitting each transformer will be printed as it is completed.	False
	verbose_feature_names_out verbose_feature_names_out: bool, str or Callable[[str, str], str], default=True - If True, :meth:`ColumnTransformer.get_feature_names_out` will prefix all feature names with the name of the transformer that generated that feature. It is equivalent to setting `verbose_feature_names_out="{transformer_name}__{feature_name}"`. - If False, :meth:`ColumnTransformer.get_feature_names_out` will not prefix any feature names and will error if feature names are not unique. - If ``Callable[[str, str], str]``, :meth:`ColumnTransformer.get_feature_names_out` will rename all the features using the name of the transformer. The first argument of the callable is the transformer name and the second argument is the feature name. The returned string will be the new feature name. - If ``str``, it must be a string ready for formatting. The given string will be formatted using two field names: ``transformer_name`` and ``feature_name``. e.g. ``"{feature_name}__{transformer_name}"``. See :meth:`str.format` method from the standard library for more info. .. versionadded:: 1.0 .. versionchanged:: 1.6 `verbose_feature_names_out` can be a callable or a string to be formatted.	True
	force_int_remainder_cols force_int_remainder_cols: bool, default=False This parameter has no effect. .. note:: If you do not access the list of columns for the remainder columns in the `transformers_` fitted attribute, you do not need to set this parameter. .. versionadded:: 1.5 .. versionchanged:: 1.7 The default value for `force_int_remainder_cols` will change from `True` to `False` in version 1.7. .. deprecated:: 1.7 `force_int_remainder_cols` is deprecated and will be removed in 1.9.	'deprecated'

	categories categories: 'auto' or a list of array-like, default='auto' Categories (unique values) per feature: - 'auto' : Determine categories automatically from the training data. - list : ``categories[i]`` holds the categories expected in the ith column. The passed categories should not mix strings and numeric values within a single feature, and should be sorted in case of numeric values. The used categories can be found in the ``categories_`` attribute. .. versionadded:: 0.20	'auto'
	drop drop: {'first', 'if_binary'} or an array-like of shape (n_features,), default=None Specifies a methodology to use to drop one of the categories per feature. This is useful in situations where perfectly collinear features cause problems, such as when feeding the resulting data into an unregularized linear regression model. However, dropping one category breaks the symmetry of the original representation and can therefore induce a bias in downstream models, for instance for penalized linear classification or regression models. - None : retain all features (the default). - 'first' : drop the first category in each feature. If only one category is present, the feature will be dropped entirely. - 'if_binary' : drop the first category in each feature with two categories. Features with 1 or more than 2 categories are left intact. - array : ``drop[i]`` is the category in feature ``X[:, i]`` that should be dropped. When `max_categories` or `min_frequency` is configured to group infrequent categories, the dropping behavior is handled after the grouping. .. versionadded:: 0.21 The parameter `drop` was added in 0.21. .. versionchanged:: 0.23 The option `drop='if_binary'` was added in 0.23. .. versionchanged:: 1.1 Support for dropping infrequent categories.	None
	sparse_output sparse_output: bool, default=True When ``True``, it returns a :class:`scipy.sparse.csr_matrix`, i.e. a sparse matrix in "Compressed Sparse Row" (CSR) format. .. versionadded:: 1.2 `sparse` was renamed to `sparse_output`	False
	dtype dtype: number type, default=np.float64 Desired dtype of output.	<class 'numpy.float64'>
	handle_unknown handle_unknown: {'error', 'ignore', 'infrequent_if_exist', 'warn'}, default='error' Specifies the way unknown categories are handled during :meth:`transform`. - 'error' : Raise an error if an unknown category is present during transform. - 'ignore' : When an unknown category is encountered during transform, the resulting one-hot encoded columns for this feature will be all zeros. In the inverse transform, an unknown category will be denoted as None. - 'infrequent_if_exist' : When an unknown category is encountered during transform, the resulting one-hot encoded columns for this feature will map to the infrequent category if it exists. The infrequent category will be mapped to the last position in the encoding. During inverse transform, an unknown category will be mapped to the category denoted `'infrequent'` if it exists. If the `'infrequent'` category does not exist, then :meth:`transform` and :meth:`inverse_transform` will handle an unknown category as with `handle_unknown='ignore'`. Infrequent categories exist based on `min_frequency` and `max_categories`. Read more in the :ref:`User Guide `. - 'warn' : When an unknown category is encountered during transform a warning is issued, and the encoding then proceeds as described for `handle_unknown="infrequent_if_exist"`. .. versionchanged:: 1.1 `'infrequent_if_exist'` was added to automatically handle unknown categories and infrequent categories. .. versionadded:: 1.6 The option `"warn"` was added in 1.6.	'error'
	min_frequency min_frequency: int or float, default=None Specifies the minimum frequency below which a category will be considered infrequent. - If `int`, categories with a smaller cardinality will be considered infrequent. - If `float`, categories with a smaller cardinality than `min_frequency * n_samples` will be considered infrequent. .. versionadded:: 1.1 Read more in the :ref:`User Guide `.	None
	max_categories max_categories: int, default=None Specifies an upper limit to the number of output features for each input feature when considering infrequent categories. If there are infrequent categories, `max_categories` includes the category representing the infrequent categories along with the frequent categories. If `None`, there is no limit to the number of output features. .. versionadded:: 1.1 Read more in the :ref:`User Guide `.	None
	feature_name_combiner feature_name_combiner: "concat" or callable, default="concat" Callable with signature `def callable(input_feature, category)` that returns a string. This is used to create feature names to be returned by :meth:`get_feature_names_out`. `"concat"` concatenates encoded feature name and category with `feature + "_" + str(category)`.E.g. feature X with values 1, 6, 7 create feature names `X_1, X_6, X_7`. .. versionadded:: 1.3	'concat'

	copy copy: bool, default=True If False, try to avoid a copy and do inplace scaling instead. This is not guaranteed to always work inplace; e.g. if the data is not a NumPy array or scipy.sparse CSR matrix, a copy may still be returned.	True
	with_mean with_mean: bool, default=True If True, center the data before scaling. This does not work (and will raise an exception) when attempted on sparse matrices, because centering them entails building a dense matrix which in common use cases is likely to be too large to fit in memory.	True
	with_std with_std: bool, default=True If True, scale the data to unit variance (or equivalently, unit standard deviation).	True

Dummy Variables and Column Transformers

Activity 3.1

Fill vs Color for Scatterplots

Nicely Formatted Axis Labels

Displaying Every Point

Redundant Legends

Removing Your Legend

Adding Descriptive Labels to Facets

geom_line()

The story last week…

Distances

Scaling

Recall: AMES Housing data

Distances and Categorical Variables

What about categorical variables?

Converting to Binary

Dummy Variables

Now we can do math!

Calculating Distances

Looking back…

Let’s reset the dataset now…

Dummifying Variables

Dummifying Variables

Dummifying in Pandas

Dummifying in Pandas

Dummifying in sklearn

Dummifying in sklearn

Column Transformers

Preprocessing

Lecture 4.1 Quiz

Preprocessing in sklearn

Column Transformers – Specify

Column Transformers – Fit

Column Transformers – Transform

Things to notice…

Lecture Activity 4.2

Multiple Preprocessing Steps

Fit!

Transform!

Finding All Categorical Variables

Fit AND Transform!

Think about it

Takeaways

Takeaways

`geom_line()`

Preprocessing in `sklearn`