API - Preprocessing¶
We provide abundant data augmentation and processing functions by using Numpy, Scipy, Threading and Queue.
However, we recommend you to use TensorFlow operation function like tf.image.central_crop
,
more TensorFlow data augmentation method can be found
here and tutorial_cifar10_tfrecord.py
.
Some of the code in this package are borrowed from Keras.
threading_data ([data, fn]) |
Return a batch of result by given data. |
rotation (x[, rg, is_random, row_index, …]) |
Rotate an image randomly or non-randomly. |
rotation_multi (x[, rg, is_random, …]) |
Rotate multiple images with the same arguments, randomly or non-randomly. |
crop (x, wrg, hrg[, is_random, row_index, …]) |
Randomly or centrally crop an image. |
crop_multi (x, wrg, hrg[, is_random, …]) |
Randomly or centrally crop multiple images. |
flip_axis (x, axis[, is_random]) |
Flip the axis of an image, such as flip left and right, up and down, randomly or non-randomly, |
flip_axis_multi (x, axis[, is_random]) |
Flip the axises of multiple images together, such as flip left and right, up and down, randomly or non-randomly, |
shift (x[, wrg, hrg, is_random, row_index, …]) |
Shift an image randomly or non-randomly. |
shift_multi (x[, wrg, hrg, is_random, …]) |
Shift images with the same arguments, randomly or non-randomly. |
shear (x[, intensity, is_random, row_index, …]) |
Shear an image randomly or non-randomly. |
shear_multi (x[, intensity, is_random, …]) |
Shear images with the same arguments, randomly or non-randomly. |
swirl (x[, center, strength, radius, …]) |
Swirl an image randomly or non-randomly, see scikit-image swirl API and example. |
swirl_multi (x[, center, strength, radius, …]) |
Swirl multiple images with the same arguments, randomly or non-randomly. |
elastic_transform (x, alpha, sigma[, mode, …]) |
Elastic deformation of images as described in [Simard2003] . |
elastic_transform_multi (x, alpha, sigma[, …]) |
Elastic deformation of images as described in [Simard2003]. |
zoom (x[, zoom_range, is_random, row_index, …]) |
Zoom in and out of a single image, randomly or non-randomly. |
zoom_multi (x[, zoom_range, is_random, …]) |
Zoom in and out of images with the same arguments, randomly or non-randomly. |
brightness (x[, gamma, gain, is_random]) |
Change the brightness of a single image, randomly or non-randomly. |
brightness_multi (x[, gamma, gain, is_random]) |
Change the brightness of multiply images, randomly or non-randomly. |
imresize (x[, size, interp, mode]) |
Resize an image by given output size and method. |
samplewise_norm (x[, rescale, …]) |
Normalize an image by rescale, samplewise centering and samplewise centering in order. |
featurewise_norm (x[, mean, std, epsilon]) |
Normalize every pixels by the same given mean and std, which are usually compute from all examples. |
channel_shift (x, intensity[, is_random, …]) |
Shift the channels of an image, randomly or non-randomly, see numpy.rollaxis. |
channel_shift_multi (x, intensity[, …]) |
Shift the channels of images with the same arguments, randomly or non-randomly, see numpy.rollaxis . |
drop (x[, keep]) |
Randomly set some pixels to zero by a given keeping probability. |
transform_matrix_offset_center (matrix, x, y) |
Return transform matrix offset center. |
apply_transform (x, transform_matrix[, …]) |
Return transformed images by given transform_matrix from transform_matrix_offset_center . |
projective_transform_by_points (x, src, dst) |
Projective transform by given coordinates, usually 4 coordinates. |
array_to_img (x[, dim_ordering, scale]) |
Converts a numpy array to PIL image object (uint8 format). |
pad_sequences (sequences[, maxlen, dtype, …]) |
Pads each sequence to the same length: the length of the longest sequence. |
precess_sequences |
|
sequences_add_start_id (sequences[, …]) |
Add special start token(id) in the beginning of each sequence. |
sequences_get_mask (sequences[, pad_val]) |
Return mask for sequences. |
distorted_images ([images, height, width]) |
Distort images for generating more training data. |
crop_central_whiten_images ([images, height, …]) |
Crop the central of image, and normailize it for test data. |
Threading¶
-
tensorlayer.prepro.
threading_data
(data=None, fn=None, **kwargs)[source]¶ Return a batch of result by given data. Usually be used for data augmentation.
Parameters: - data : numpy array or zip of numpy array, see Examples below.
- fn : the function for data processing.
- more args : the args for fn, see Examples below.
References
Examples
- Single array
>>> X --> [batch_size, row, col, 1] greyscale >>> results = threading_data(X, zoom, zoom_range=[0.5, 1], is_random=True) ... results --> [batch_size, row, col, channel] >>> tl.visualize.images2d(images=np.asarray(results), second=0.01, saveable=True, name='after', dtype=None) >>> tl.visualize.images2d(images=np.asarray(X), second=0.01, saveable=True, name='before', dtype=None)
- List of array (e.g. functions with
multi
)
>>> X, Y --> [batch_size, row, col, 1] greyscale >>> data = threading_data([_ for _ in zip(X, Y)], zoom_multi, zoom_range=[0.5, 1], is_random=True) ... data --> [batch_size, 2, row, col, 1] >>> X_, Y_ = data.transpose((1,0,2,3,4)) ... X_, Y_ --> [batch_size, row, col, 1] >>> tl.visualize.images2d(images=np.asarray(X_), second=0.01, saveable=True, name='after', dtype=None) >>> tl.visualize.images2d(images=np.asarray(Y_), second=0.01, saveable=True, name='before', dtype=None)
- Customized function for image segmentation
>>> def distort_img(data): ... x, y = data ... x, y = flip_axis_multi([x, y], axis=0, is_random=True) ... x, y = flip_axis_multi([x, y], axis=1, is_random=True) ... x, y = crop_multi([x, y], 100, 100, is_random=True) ... return x, y >>> X, Y --> [batch_size, row, col, channel] >>> data = threading_data([_ for _ in zip(X, Y)], distort_img) >>> X_, Y_ = data.transpose((1,0,2,3,4))
Images¶
- These functions only apply on a single image, use
threading_data
to apply multiple threading seetutorial_image_preprocess.py
. - All functions have argument
is_random
. - All functions end with multi , usually be used for image segmentation i.e. the input and output image should be matched.
Rotation¶
-
tensorlayer.prepro.
rotation
(x, rg=20, is_random=False, row_index=0, col_index=1, channel_index=2, fill_mode='nearest', cval=0.0)[source]¶ Rotate an image randomly or non-randomly.
Parameters: - x : numpy array
An image with dimension of [row, col, channel] (default).
- rg : int or float
Degree to rotate, usually 0 ~ 180.
- is_random : boolean, default False
If True, randomly rotate.
- row_index, col_index, channel_index : int
Index of row, col and channel, default (0, 1, 2), for theano (1, 2, 0).
- fill_mode : string
Method to fill missing pixel, default ‘nearest’, more options ‘constant’, ‘reflect’ or ‘wrap’
- cval : scalar, optional
Value used for points outside the boundaries of the input if mode=’constant’. Default is 0.0
Examples
>>> x --> [row, col, 1] greyscale >>> x = rotation(x, rg=40, is_random=False) >>> tl.visualize.frame(x[:,:,0], second=0.01, saveable=True, name='temp',cmap='gray')
-
tensorlayer.prepro.
rotation_multi
(x, rg=20, is_random=False, row_index=0, col_index=1, channel_index=2, fill_mode='nearest', cval=0.0)[source]¶ Rotate multiple images with the same arguments, randomly or non-randomly. Usually be used for image segmentation which x=[X, Y], X and Y should be matched.
Parameters: - x : list of numpy array
List of images with dimension of [n_images, row, col, channel] (default).
- others : see
rotation
.
Examples
>>> x, y --> [row, col, 1] greyscale >>> x, y = rotation_multi([x, y], rg=90, is_random=False) >>> tl.visualize.frame(x[:,:,0], second=0.01, saveable=True, name='x',cmap='gray') >>> tl.visualize.frame(y[:,:,0], second=0.01, saveable=True, name='y',cmap='gray')
Crop¶
-
tensorlayer.prepro.
crop
(x, wrg, hrg, is_random=False, row_index=0, col_index=1, channel_index=2)[source]¶ Randomly or centrally crop an image.
Parameters: - x : numpy array
An image with dimension of [row, col, channel] (default).
- wrg : float
Size of weight.
- hrg : float
Size of height.
- is_random : boolean, default False
If True, randomly crop, else central crop.
- row_index, col_index, channel_index : int
Index of row, col and channel, default (0, 1, 2), for theano (1, 2, 0).
Flip¶
-
tensorlayer.prepro.
flip_axis
(x, axis, is_random=False)[source]¶ Flip the axis of an image, such as flip left and right, up and down, randomly or non-randomly,
Parameters: - x : numpy array
An image with dimension of [row, col, channel] (default).
- axis : int
- 0, flip up and down
- 1, flip left and right
- 2, flip channel
- is_random : boolean, default False
If True, randomly zoom.
-
tensorlayer.prepro.
flip_axis_multi
(x, axis, is_random=False)[source]¶ Flip the axises of multiple images together, such as flip left and right, up and down, randomly or non-randomly,
Parameters: - x : list of numpy array
List of images with dimension of [n_images, row, col, channel] (default).
- others : see
flip_axis
.
Shift¶
-
tensorlayer.prepro.
shift
(x, wrg=0.1, hrg=0.1, is_random=False, row_index=0, col_index=1, channel_index=2, fill_mode='nearest', cval=0.0)[source]¶ Shift an image randomly or non-randomly.
Parameters: - x : numpy array
An image with dimension of [row, col, channel] (default).
- wrg : float
Percentage of shift in axis x, usually -0.25 ~ 0.25.
- hrg : float
Percentage of shift in axis y, usually -0.25 ~ 0.25.
- is_random : boolean, default False
If True, randomly shift.
- row_index, col_index, channel_index : int
Index of row, col and channel, default (0, 1, 2), for theano (1, 2, 0).
- fill_mode : string
Method to fill missing pixel, default ‘nearest’, more options ‘constant’, ‘reflect’ or ‘wrap’.
- cval : scalar, optional
Value used for points outside the boundaries of the input if mode=’constant’. Default is 0.0.
-
tensorlayer.prepro.
shift_multi
(x, wrg=0.1, hrg=0.1, is_random=False, row_index=0, col_index=1, channel_index=2, fill_mode='nearest', cval=0.0)[source]¶ Shift images with the same arguments, randomly or non-randomly. Usually be used for image segmentation which x=[X, Y], X and Y should be matched.
Parameters: - x : list of numpy array
List of images with dimension of [n_images, row, col, channel] (default).
- others : see
shift
.
Shear¶
-
tensorlayer.prepro.
shear
(x, intensity=0.1, is_random=False, row_index=0, col_index=1, channel_index=2, fill_mode='nearest', cval=0.0)[source]¶ Shear an image randomly or non-randomly.
Parameters: - x : numpy array
An image with dimension of [row, col, channel] (default).
- intensity : float
Percentage of shear, usually -0.5 ~ 0.5 (is_random==True), 0 ~ 0.5 (is_random==False), you can have a quick try by shear(X, 1).
- is_random : boolean, default False
If True, randomly shear.
- row_index, col_index, channel_index : int
Index of row, col and channel, default (0, 1, 2), for theano (1, 2, 0).
- fill_mode : string
Method to fill missing pixel, default ‘nearest’, more options ‘constant’, ‘reflect’ or ‘wrap’.
- cval : scalar, optional
Value used for points outside the boundaries of the input if mode=’constant’. Default is 0.0.
-
tensorlayer.prepro.
shear_multi
(x, intensity=0.1, is_random=False, row_index=0, col_index=1, channel_index=2, fill_mode='nearest', cval=0.0)[source]¶ Shear images with the same arguments, randomly or non-randomly. Usually be used for image segmentation which x=[X, Y], X and Y should be matched.
Parameters: - x : list of numpy array
List of images with dimension of [n_images, row, col, channel] (default).
- others : see
shear
.
Swirl¶
-
tensorlayer.prepro.
swirl
(x, center=None, strength=1, radius=100, rotation=0, output_shape=None, order=1, mode='constant', cval=0, clip=True, preserve_range=False, is_random=False)[source]¶ Swirl an image randomly or non-randomly, see scikit-image swirl API and example.
Parameters: - x : numpy array
An image with dimension of [row, col, channel] (default).
- center : (row, column) tuple or (2,) ndarray, optional
Center coordinate of transformation.
- strength : float, optional
The amount of swirling applied.
- radius : float, optional
The extent of the swirl in pixels. The effect dies out rapidly beyond radius.
- rotation : float, (degree) optional
Additional rotation applied to the image, usually [0, 360], relates to center.
- output_shape : tuple (rows, cols), optional
Shape of the output image generated. By default the shape of the input image is preserved.
- order : int, optional
The order of the spline interpolation, default is 1. The order has to be in the range 0-5. See skimage.transform.warp for detail.
- mode : {‘constant’, ‘edge’, ‘symmetric’, ‘reflect’, ‘wrap’}, optional
Points outside the boundaries of the input are filled according to the given mode, with ‘constant’ used as the default. Modes match the behaviour of numpy.pad.
- cval : float, optional
Used in conjunction with mode ‘constant’, the value outside the image boundaries.
- clip : bool, optional
Whether to clip the output to the range of values of the input image. This is enabled by default, since higher order interpolation may produce values outside the given input range.
- preserve_range : bool, optional
Whether to keep the original range of values. Otherwise, the input image is converted according to the conventions of img_as_float.
- is_random : boolean, default False
- If True, random swirl.
- random center = [(0 ~ x.shape[0]), (0 ~ x.shape[1])]
- random strength = [0, strength]
- random radius = [1e-10, radius]
- random rotation = [-rotation, rotation]
Examples
>>> x --> [row, col, 1] greyscale >>> x = swirl(x, strength=4, radius=100)
-
tensorlayer.prepro.
swirl_multi
(x, center=None, strength=1, radius=100, rotation=0, output_shape=None, order=1, mode='constant', cval=0, clip=True, preserve_range=False, is_random=False)[source]¶ Swirl multiple images with the same arguments, randomly or non-randomly. Usually be used for image segmentation which x=[X, Y], X and Y should be matched.
Parameters: - x : list of numpy array
List of images with dimension of [n_images, row, col, channel] (default).
- others : see
swirl
.
Elastic transform¶
-
tensorlayer.prepro.
elastic_transform
(x, alpha, sigma, mode='constant', cval=0, is_random=False)[source]¶ Elastic deformation of images as described in [Simard2003] .
Parameters: - x : numpy array, a greyscale image.
- alpha : scalar factor.
- sigma : scalar or sequence of scalars, the smaller the sigma, the more transformation.
Standard deviation for Gaussian kernel. The standard deviations of the Gaussian filter are given for each axis as a sequence, or as a single number, in which case it is equal for all axes.
- mode : default constant, see scipy.ndimage.filters.gaussian_filter.
- cval : float, optional. Used in conjunction with mode ‘constant’, the value outside the image boundaries.
- is_random : boolean, default False
References
Examples
>>> x = elastic_transform(x, alpha = x.shape[1] * 3, sigma = x.shape[1] * 0.07)
-
tensorlayer.prepro.
elastic_transform_multi
(x, alpha, sigma, mode='constant', cval=0, is_random=False)[source]¶ Elastic deformation of images as described in [Simard2003].
Parameters: - x : list of numpy array
- others : see
elastic_transform
.
Zoom¶
-
tensorlayer.prepro.
zoom
(x, zoom_range=(0.9, 1.1), is_random=False, row_index=0, col_index=1, channel_index=2, fill_mode='nearest', cval=0.0)[source]¶ Zoom in and out of a single image, randomly or non-randomly.
Parameters: - x : numpy array
An image with dimension of [row, col, channel] (default).
- zoom_range : list or tuple
- If is_random=False, (h, w) are the fixed zoom factor for row and column axies, factor small than one is zoom in.
- If is_random=True, (min zoom out, max zoom out) for x and y with different random zoom in/out factor.
e.g (0.5, 1) zoom in 1~2 times.
- is_random : boolean, default False
If True, randomly zoom.
- row_index, col_index, channel_index : int
Index of row, col and channel, default (0, 1, 2), for theano (1, 2, 0).
- fill_mode : string
Method to fill missing pixel, default ‘nearest’, more options ‘constant’, ‘reflect’ or ‘wrap’.
- cval : scalar, optional
Value used for points outside the boundaries of the input if mode=’constant’. Default is 0.0.
-
tensorlayer.prepro.
zoom_multi
(x, zoom_range=(0.9, 1.1), is_random=False, row_index=0, col_index=1, channel_index=2, fill_mode='nearest', cval=0.0)[source]¶ Zoom in and out of images with the same arguments, randomly or non-randomly. Usually be used for image segmentation which x=[X, Y], X and Y should be matched.
Parameters: - x : list of numpy array
List of images with dimension of [n_images, row, col, channel] (default).
- others : see
zoom
.
Brightness¶
-
tensorlayer.prepro.
brightness
(x, gamma=1, gain=1, is_random=False)[source]¶ Change the brightness of a single image, randomly or non-randomly.
Parameters: - x : numpy array
An image with dimension of [row, col, channel] (default).
- gamma : float, small than 1 means brighter.
Non negative real number. Default value is 1.
- If is_random is True, gamma in a range of (1-gamma, 1+gamma).
- gain : float
The constant multiplier. Default value is 1.
- is_random : boolean, default False
- If True, randomly change brightness.
References
-
tensorlayer.prepro.
brightness_multi
(x, gamma=1, gain=1, is_random=False)[source]¶ Change the brightness of multiply images, randomly or non-randomly. Usually be used for image segmentation which x=[X, Y], X and Y should be matched.
Parameters: - x : list of numpy array
List of images with dimension of [n_images, row, col, channel] (default).
- others : see
brightness
.
Resize¶
-
tensorlayer.prepro.
imresize
(x, size=[100, 100], interp='bilinear', mode=None)[source]¶ Resize an image by given output size and method. Warning, this function will rescale the value to [0, 255].
Parameters: - x : numpy array
An image with dimension of [row, col, channel] (default).
- size : int, float or tuple (h, w)
- int, Percentage of current size.
- float, Fraction of current size.
- tuple, Size of the output image.
- interp : str, optional
Interpolation to use for re-sizing (‘nearest’, ‘lanczos’, ‘bilinear’, ‘bicubic’ or ‘cubic’).
- mode : str, optional
The PIL image mode (‘P’, ‘L’, etc.) to convert arr before resizing.
Returns: - imresize : ndarray
- The resized array of image.
References
Normalization¶
-
tensorlayer.prepro.
samplewise_norm
(x, rescale=None, samplewise_center=False, samplewise_std_normalization=False, channel_index=2, epsilon=1e-07)[source]¶ Normalize an image by rescale, samplewise centering and samplewise centering in order.
Parameters: - x : numpy array
An image with dimension of [row, col, channel] (default).
- rescale : rescaling factor.
If None or 0, no rescaling is applied, otherwise we multiply the data by the value provided (before applying any other transformation)
- samplewise_center : set each sample mean to 0.
- samplewise_std_normalization : divide each input by its std.
- epsilon : small position value for dividing standard deviation.
Notes
When samplewise_center and samplewise_std_normalization are True.
- For greyscale image, every pixels are subtracted and divided by the mean and std of whole image.
- For RGB image, every pixels are subtracted and divided by the mean and std of this pixel i.e. the mean and std of a pixel is 0 and 1.
Examples
>>> x = samplewise_norm(x, samplewise_center=True, samplewise_std_normalization=True) >>> print(x.shape, np.mean(x), np.std(x)) ... (160, 176, 1), 0.0, 1.0
-
tensorlayer.prepro.
featurewise_norm
(x, mean=None, std=None, epsilon=1e-07)[source]¶ Normalize every pixels by the same given mean and std, which are usually compute from all examples.
Parameters: - x : numpy array
An image with dimension of [row, col, channel] (default).
- mean : value for subtraction.
- std : value for division.
- epsilon : small position value for dividing standard deviation.
Channel shift¶
-
tensorlayer.prepro.
channel_shift
(x, intensity, is_random=False, channel_index=2)[source]¶ Shift the channels of an image, randomly or non-randomly, see numpy.rollaxis.
Parameters: - x : numpy array
An image with dimension of [row, col, channel] (default).
- intensity : float
Intensity of shifting.
- is_random : boolean, default False
If True, randomly shift.
- channel_index : int
Index of channel, default 2.
-
tensorlayer.prepro.
channel_shift_multi
(x, intensity, channel_index=2)[source]¶ Shift the channels of images with the same arguments, randomly or non-randomly, see numpy.rollaxis . Usually be used for image segmentation which x=[X, Y], X and Y should be matched.
Parameters: - x : list of numpy array
List of images with dimension of [n_images, row, col, channel] (default).
- others : see
channel_shift
.
Noise¶
Manual transform¶
-
tensorlayer.prepro.
transform_matrix_offset_center
(matrix, x, y)[source]¶ Return transform matrix offset center.
Parameters: - matrix : numpy array
Transform matrix
- x, y : int
Size of image.
Examples
- See
rotation
,shear
,zoom
.
-
tensorlayer.prepro.
apply_transform
(x, transform_matrix, channel_index=2, fill_mode='nearest', cval=0.0)[source]¶ Return transformed images by given transform_matrix from
transform_matrix_offset_center
.Parameters: - x : numpy array
Batch of images with dimension of 3, [batch_size, row, col, channel].
- transform_matrix : numpy array
Transform matrix (offset center), can be generated by
transform_matrix_offset_center
- channel_index : int
Index of channel, default 2.
- fill_mode : string
Method to fill missing pixel, default ‘nearest’, more options ‘constant’, ‘reflect’ or ‘wrap’
- cval : scalar, optional
Value used for points outside the boundaries of the input if mode=’constant’. Default is 0.0
Examples
- See
rotation
,shift
,shear
,zoom
.
-
tensorlayer.prepro.
projective_transform_by_points
(x, src, dst, map_args={}, output_shape=None, order=1, mode='constant', cval=0.0, clip=True, preserve_range=False)[source]¶ Projective transform by given coordinates, usually 4 coordinates. see scikit-image.
Parameters: - x : numpy array
An image with dimension of [row, col, channel] (default).
- src : list or numpy
The original coordinates, usually 4 coordinates of (x, y).
- dst : list or numpy
The coordinates after transformation, the number of coordinates is the same with src.
- map_args : dict, optional
Keyword arguments passed to inverse_map.
- output_shape : tuple (rows, cols), optional
Shape of the output image generated. By default the shape of the input image is preserved. Note that, even for multi-band images, only rows and columns need to be specified.
- order : int, optional
The order of interpolation. The order has to be in the range 0-5:
- 0 Nearest-neighbor
- 1 Bi-linear (default)
- 2 Bi-quadratic
- 3 Bi-cubic
- 4 Bi-quartic
- 5 Bi-quintic
- mode : {‘constant’, ‘edge’, ‘symmetric’, ‘reflect’, ‘wrap’}, optional
Points outside the boundaries of the input are filled according to the given mode. Modes match the behaviour of numpy.pad.
- cval : float, optional
Used in conjunction with mode ‘constant’, the value outside the image boundaries.
- clip : bool, optional
Whether to clip the output to the range of values of the input image. This is enabled by default, since higher order interpolation may produce values outside the given input range.
- preserve_range : bool, optional
Whether to keep the original range of values. Otherwise, the input image is converted according to the conventions of img_as_float.
References
Examples
>>> Assume X is an image from CIFAR 10, i.e. shape == (32, 32, 3) >>> src = [[0,0],[0,32],[32,0],[32,32]] >>> dst = [[10,10],[0,32],[32,0],[32,32]] >>> x = projective_transform_by_points(X, src, dst)
Numpy and PIL¶
-
tensorlayer.prepro.
array_to_img
(x, dim_ordering=(0, 1, 2), scale=True)[source]¶ Converts a numpy array to PIL image object (uint8 format).
Parameters: - x : numpy array
A image with dimension of 3 and channels of 1 or 3.
- dim_ordering : list or tuple of 3 int
Index of row, col and channel, default (0, 1, 2), for theano (1, 2, 0).
- scale : boolean, default is True
If True, converts image to [0, 255] from any range of value like [-1, 2].
References
Sequence¶
More related functions can be found in tensorlayer.nlp
.
Padding¶
-
tensorlayer.prepro.
pad_sequences
(sequences, maxlen=None, dtype='int32', padding='post', truncating='pre', value=0.0)[source]¶ Pads each sequence to the same length: the length of the longest sequence. If maxlen is provided, any sequence longer than maxlen is truncated to maxlen. Truncation happens off either the beginning (default) or the end of the sequence. Supports post-padding and pre-padding (default).
Parameters: - sequences : list of lists where each element is a sequence
- maxlen : int, maximum length
- dtype : type to cast the resulting sequence.
- padding : ‘pre’ or ‘post’, pad either before or after each sequence.
- truncating : ‘pre’ or ‘post’, remove values from sequences larger than
maxlen either in the beginning or in the end of the sequence
- value : float, value to pad the sequences to the desired value.
Returns: - x : numpy array with dimensions (number_of_sequences, maxlen)
Examples
>>> sequences = [[1,1,1,1,1],[2,2,2],[3,3]] >>> sequences = pad_sequences(sequences, maxlen=None, dtype='int32', ... padding='post', truncating='pre', value=0.) ... [[1 1 1 1 1] ... [2 2 2 0 0] ... [3 3 0 0 0]]
Process¶
-
tensorlayer.prepro.
process_sequences
(sequences, end_id=0, pad_val=0, is_shorten=True, remain_end_id=False)[source]¶ Set all tokens(ids) after END token to the padding value, and then shorten (option) it to the maximum sequence length in this batch.
Parameters: - sequences : numpy array or list of list with token IDs.
e.g. [[4,3,5,3,2,2,2,2], [5,3,9,4,9,2,2,3]]
- end_id : int, the special token for END.
- pad_val : int, replace the end_id and the ids after end_id to this value.
- is_shorten : boolean, default True.
Shorten the sequences.
- remain_end_id : boolean, default False.
Keep an end_id in the end.
Examples
>>> sentences_ids = [[4, 3, 5, 3, 2, 2, 2, 2], <-- end_id is 2 ... [5, 3, 9, 4, 9, 2, 2, 3]] <-- end_id is 2 >>> sentences_ids = precess_sequences(sentences_ids, end_id=vocab.end_id, pad_val=0, is_shorten=True) ... [[4, 3, 5, 3, 0], [5, 3, 9, 4, 9]]
Add Start ID¶
-
tensorlayer.prepro.
sequences_add_start_id
(sequences, start_id=0, remove_last=False)[source]¶ Add special start token(id) in the beginning of each sequence.
Examples
>>> sentences_ids = [[4,3,5,3,2,2,2,2], [5,3,9,4,9,2,2,3]] >>> sentences_ids = sequences_add_start_id(sentences_ids, start_id=2) ... [[2, 4, 3, 5, 3, 2, 2, 2, 2], [2, 5, 3, 9, 4, 9, 2, 2, 3]] >>> sentences_ids = sequences_add_start_id(sentences_ids, start_id=2, remove_last=True) ... [[2, 4, 3, 5, 3, 2, 2, 2], [2, 5, 3, 9, 4, 9, 2, 2]]
- For Seq2seq
>>> input = [a, b, c] >>> target = [x, y, z] >>> decode_seq = [start_id, a, b] <-- sequences_add_start_id(input, start_id, True)
Tensor Opt¶
Note
These functions will be deprecated, see tutorial_cifar10_tfrecord.py
for new information.
-
tensorlayer.prepro.
distorted_images
(images=None, height=24, width=24)[source]¶ Distort images for generating more training data.
Parameters: - images : 4D Tensor
The tensor or placeholder of images
- height : int
The height for random crop.
- width : int
The width for random crop.
Returns: - result : tuple of Tensor
(Tensor for distorted images, Tensor for while loop index)
Notes
- The first image in ‘distorted_images’ should be removed.
References
Examples
>>> X_train, y_train, X_test, y_test = tl.files.load_cifar10_dataset(shape=(-1, 32, 32, 3), plotable=False) >>> sess = tf.InteractiveSession() >>> batch_size = 128 >>> x = tf.placeholder(tf.float32, shape=[batch_size, 32, 32, 3]) >>> distorted_images_op = tl.preprocess.distorted_images(images=x, height=24, width=24) >>> sess.run(tf.initialize_all_variables()) >>> feed_dict={x: X_train[0:batch_size,:,:,:]} >>> distorted_images, idx = sess.run(distorted_images_op, feed_dict=feed_dict) >>> tl.visualize.images2d(X_train[0:9,:,:,:], second=2, saveable=False, name='cifar10', dtype=np.uint8, fig_idx=20212) >>> tl.visualize.images2d(distorted_images[1:10,:,:,:], second=10, saveable=False, name='distorted_images', dtype=None, fig_idx=23012)
-
tensorlayer.prepro.
crop_central_whiten_images
(images=None, height=24, width=24)[source]¶ Crop the central of image, and normailize it for test data.
They are cropped to central of height * width pixels.
Whiten (Normalize) the images.
Parameters: - images : 4D Tensor
The tensor or placeholder of images
- height : int
The height for central crop.
- width : int
The width for central crop.
Returns: - result : tuple Tensor
(Tensor for distorted images, Tensor for while loop index)
Notes
The first image in ‘central_images’ should be removed.
Examples
>>> X_train, y_train, X_test, y_test = tl.files.load_cifar10_dataset(shape=(-1, 32, 32, 3), plotable=False) >>> sess = tf.InteractiveSession() >>> batch_size = 128 >>> x = tf.placeholder(tf.float32, shape=[batch_size, 32, 32, 3]) >>> central_images_op = tl.preprocess.crop_central_whiten_images(images=x, height=24, width=24) >>> sess.run(tf.initialize_all_variables()) >>> feed_dict={x: X_train[0:batch_size,:,:,:]} >>> central_images, idx = sess.run(central_images_op, feed_dict=feed_dict) >>> tl.visualize.images2d(X_train[0:9,:,:,:], second=2, saveable=False, name='cifar10', dtype=np.uint8, fig_idx=20212) >>> tl.visualize.images2d(central_images[1:10,:,:,:], second=10, saveable=False, name='central_images', dtype=None, fig_idx=23012)