Skip to the stories
AI & ML

NumPy Views vs. Copies: Preventing Silent Changes in ML Preprocessing

  • Sohel Alam
  • 702 reads

NumPy slices can share memory with their source, so in-place preprocessing may alter data unexpectedly. Learn when to copy and how to check.

NumPy Views vs. Copies: Preventing Silent Changes in ML Preprocessing

In NumPy preprocessing, a new variable doesn’t necessarily mean new data. Basic slicing often creates a view that shares memory with its source, so an in-place operation on selected features can also change the original dataset. The result may be a pipeline that runs cleanly while later steps use altered inputs.

Views and copies behave differently

A view is a separate array object that refers to data held by another array. A copy allocates independent data. They can display the same values, but changes to a view can reach its source.

import numpy as np

original = np.array([10, 20, 30, 40, 50])
view = original[1:4]
view[0] = 999

print(original)  # [ 10 999  30  40  50]
print(view)      # [999  30  40]

The slice original[1:4] is a view, so assigning through view updates the shared elements. Add .copy() when the selected data needs its own storage:

original = np.array([10, 20, 30, 40, 50])
copied = original[1:4].copy()
copied[0] = 999

print(original)  # [10 20 30 40 50]
print(copied)    # [999  30  40]

The key difference is memory ownership, not the variable name or the values printed. The same distinction appears in other Python data structures, though their copying behavior has its own details.

Why preprocessing can modify the source data

Feature selection commonly uses slicing. Consider an array with three columns, where the first two are features to center around zero:

data = np.array([
    [100, 20, 5],
    [200, 30, 10],
    [300, 40, 15],
    [400, 50, 20]
], dtype=float)

features = data[:, :2]
features -= features.mean(axis=0)

The selection assigns a view of the first two columns to features. The subtraction is in place, so the selected values in data change too. The resulting feature values are centered, but the original array now contains those transformed values in its first two columns.

That can matter when later steps need the untouched dataset—for example, another preprocessing path, a model comparison, or evaluation. The code may produce plausible outputs and no exception, leaving the data change difficult to spot.

To keep the source intact, copy the selected columns before transforming them:

features = data[:, :2].copy()
features -= features.mean(axis=0)

This makes the intent explicit and isolates the preprocessing. Copying consumes additional memory, so copying the full dataset by default may be expensive for large arrays. Copy the portion that must remain independent, and check sharing when the operation’s behavior isn’t clear.

Check what shares memory

np.shares_memory() directly tests whether two arrays share any memory:

original = np.arange(10)
view = original[2:7]
copied = original[2:7].copy()

print(np.shares_memory(original, view))    # True
print(np.shares_memory(original, copied))  # False

The .base attribute can also reveal the array that owns the memory behind a view:

print(view.base is original)  # True

That attribute is useful for inspection, but it isn’t a complete memory-sharing test. Use np.shares_memory() when you need to check whether arrays overlap.

Selection and reshaping don’t all have the same behavior

NumPy’s results depend on the operation. Basic slices such as arr[1:5] return views. Advanced indexing, including Boolean masks and lists of indices, returns copies. copy() and flatten() make copies; reshape() and ravel() return views when possible and otherwise copy.

That difference can be easy to miss because basic and advanced indexing can select similar elements:

arr = np.array([10, 20, 30, 40])

selected = arr[1:3]  # basic slicing: view
selected[0] = 999
print(arr)            # [ 10 999  30  40]

arr = np.array([10, 20, 30, 40])

selected = arr[[1, 2]]  # advanced indexing: copy
selected[0] = 999
print(arr)               # [10 20 30 40]

Don’t infer memory independence from the fact that an expression returned an array. Check the operation’s behavior or test it directly.

In-place arithmetic is the trigger to watch

In-place operators such as +=, -=, and /= write into the current array. When that array is a view, the writes affect its source:

arr = np.array([10, 20, 30])
view = arr[:]
view += 5
print(arr)  # [15 25 35]

Regular arithmetic creates a result array and reassigns the variable, so the source remains unchanged:

arr = np.array([10, 20, 30])
view = arr[:]
view = view + 5
print(arr)  # [10 20 30]

Hand-written scaling code often uses in-place operations to limit extra allocations. That can be sensible for memory use, but only after confirming that the target array is safe to modify.

Keep pipeline bugs distinct from data leakage

Shared memory can cause several problems in an ML workflow. Preprocessing may alter the initial dataset, making later experiments depend on earlier mutations. If training and test arrays are views into the same source, transforming one in place can also change data visible through the other.

That shared-memory error is distinct from data leakage. Leakage occurs when information from test data influences training—for example, fitting a scaler on the full dataset before splitting. The usual sequence is to split first, fit preprocessing on training data, then apply the learned transformation to test data.

Many scikit-learn transformers, including StandardScaler, return new arrays by default. Some support a copy=False option, however, and custom NumPy code may modify views directly. A pipeline’s high-level structure doesn’t remove the need to understand what its arrays share.

A quick check before transforming

Before modifying an array, ask whether it came from basic slicing, whether it may share memory with its source, and whether the next operation writes in place. If the original must stay unchanged, make an explicit copy—of the whole array or just the selected region, depending on the workflow and memory budget.

X_processed = X_original.copy()

When unexpected values appear later, inspect how the array was created and whether the transformation mutated it. In NumPy preprocessing, one explicit copy at the right boundary can prevent a silent change from spreading through the rest of the pipeline.

Source: Sohel Alam · hackernoon.com

From the same set